{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "65dba206",
   "metadata": {},
   "source": [
    "# DAY 34\n",
    "\n",
    "首先回顾下昨天的内容,我在训练开始和结束增加了time来查看运行时长"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "50d27505",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [100/20000], Loss: 1.1143\n",
      "Epoch [200/20000], Loss: 1.0841\n",
      "Epoch [300/20000], Loss: 1.0578\n",
      "Epoch [400/20000], Loss: 1.0265\n",
      "Epoch [500/20000], Loss: 0.9851\n",
      "Epoch [600/20000], Loss: 0.9316\n",
      "Epoch [700/20000], Loss: 0.8681\n",
      "Epoch [800/20000], Loss: 0.7994\n",
      "Epoch [900/20000], Loss: 0.7325\n",
      "Epoch [1000/20000], Loss: 0.6736\n",
      "Epoch [1100/20000], Loss: 0.6241\n",
      "Epoch [1200/20000], Loss: 0.5835\n",
      "Epoch [1300/20000], Loss: 0.5502\n",
      "Epoch [1400/20000], Loss: 0.5225\n",
      "Epoch [1500/20000], Loss: 0.4992\n",
      "Epoch [1600/20000], Loss: 0.4789\n",
      "Epoch [1700/20000], Loss: 0.4609\n",
      "Epoch [1800/20000], Loss: 0.4447\n",
      "Epoch [1900/20000], Loss: 0.4299\n",
      "Epoch [2000/20000], Loss: 0.4160\n",
      "Epoch [2100/20000], Loss: 0.4029\n",
      "Epoch [2200/20000], Loss: 0.3906\n",
      "Epoch [2300/20000], Loss: 0.3787\n",
      "Epoch [2400/20000], Loss: 0.3674\n",
      "Epoch [2500/20000], Loss: 0.3564\n",
      "Epoch [2600/20000], Loss: 0.3458\n",
      "Epoch [2700/20000], Loss: 0.3355\n",
      "Epoch [2800/20000], Loss: 0.3254\n",
      "Epoch [2900/20000], Loss: 0.3157\n",
      "Epoch [3000/20000], Loss: 0.3062\n",
      "Epoch [3100/20000], Loss: 0.2970\n",
      "Epoch [3200/20000], Loss: 0.2880\n",
      "Epoch [3300/20000], Loss: 0.2794\n",
      "Epoch [3400/20000], Loss: 0.2710\n",
      "Epoch [3500/20000], Loss: 0.2628\n",
      "Epoch [3600/20000], Loss: 0.2549\n",
      "Epoch [3700/20000], Loss: 0.2472\n",
      "Epoch [3800/20000], Loss: 0.2399\n",
      "Epoch [3900/20000], Loss: 0.2329\n",
      "Epoch [4000/20000], Loss: 0.2263\n",
      "Epoch [4100/20000], Loss: 0.2199\n",
      "Epoch [4200/20000], Loss: 0.2139\n",
      "Epoch [4300/20000], Loss: 0.2082\n",
      "Epoch [4400/20000], Loss: 0.2028\n",
      "Epoch [4500/20000], Loss: 0.1977\n",
      "Epoch [4600/20000], Loss: 0.1928\n",
      "Epoch [4700/20000], Loss: 0.1881\n",
      "Epoch [4800/20000], Loss: 0.1837\n",
      "Epoch [4900/20000], Loss: 0.1795\n",
      "Epoch [5000/20000], Loss: 0.1755\n",
      "Epoch [5100/20000], Loss: 0.1716\n",
      "Epoch [5200/20000], Loss: 0.1680\n",
      "Epoch [5300/20000], Loss: 0.1645\n",
      "Epoch [5400/20000], Loss: 0.1611\n",
      "Epoch [5500/20000], Loss: 0.1580\n",
      "Epoch [5600/20000], Loss: 0.1549\n",
      "Epoch [5700/20000], Loss: 0.1520\n",
      "Epoch [5800/20000], Loss: 0.1492\n",
      "Epoch [5900/20000], Loss: 0.1465\n",
      "Epoch [6000/20000], Loss: 0.1440\n",
      "Epoch [6100/20000], Loss: 0.1415\n",
      "Epoch [6200/20000], Loss: 0.1392\n",
      "Epoch [6300/20000], Loss: 0.1369\n",
      "Epoch [6400/20000], Loss: 0.1348\n",
      "Epoch [6500/20000], Loss: 0.1327\n",
      "Epoch [6600/20000], Loss: 0.1307\n",
      "Epoch [6700/20000], Loss: 0.1287\n",
      "Epoch [6800/20000], Loss: 0.1269\n",
      "Epoch [6900/20000], Loss: 0.1251\n",
      "Epoch [7000/20000], Loss: 0.1234\n",
      "Epoch [7100/20000], Loss: 0.1217\n",
      "Epoch [7200/20000], Loss: 0.1201\n",
      "Epoch [7300/20000], Loss: 0.1186\n",
      "Epoch [7400/20000], Loss: 0.1171\n",
      "Epoch [7500/20000], Loss: 0.1157\n",
      "Epoch [7600/20000], Loss: 0.1143\n",
      "Epoch [7700/20000], Loss: 0.1129\n",
      "Epoch [7800/20000], Loss: 0.1116\n",
      "Epoch [7900/20000], Loss: 0.1104\n",
      "Epoch [8000/20000], Loss: 0.1092\n",
      "Epoch [8100/20000], Loss: 0.1080\n",
      "Epoch [8200/20000], Loss: 0.1068\n",
      "Epoch [8300/20000], Loss: 0.1057\n",
      "Epoch [8400/20000], Loss: 0.1047\n",
      "Epoch [8500/20000], Loss: 0.1036\n",
      "Epoch [8600/20000], Loss: 0.1026\n",
      "Epoch [8700/20000], Loss: 0.1017\n",
      "Epoch [8800/20000], Loss: 0.1007\n",
      "Epoch [8900/20000], Loss: 0.0998\n",
      "Epoch [9000/20000], Loss: 0.0989\n",
      "Epoch [9100/20000], Loss: 0.0980\n",
      "Epoch [9200/20000], Loss: 0.0972\n",
      "Epoch [9300/20000], Loss: 0.0964\n",
      "Epoch [9400/20000], Loss: 0.0956\n",
      "Epoch [9500/20000], Loss: 0.0948\n",
      "Epoch [9600/20000], Loss: 0.0941\n",
      "Epoch [9700/20000], Loss: 0.0933\n",
      "Epoch [9800/20000], Loss: 0.0926\n",
      "Epoch [9900/20000], Loss: 0.0919\n",
      "Epoch [10000/20000], Loss: 0.0913\n",
      "Epoch [10100/20000], Loss: 0.0906\n",
      "Epoch [10200/20000], Loss: 0.0900\n",
      "Epoch [10300/20000], Loss: 0.0893\n",
      "Epoch [10400/20000], Loss: 0.0887\n",
      "Epoch [10500/20000], Loss: 0.0881\n",
      "Epoch [10600/20000], Loss: 0.0876\n",
      "Epoch [10700/20000], Loss: 0.0870\n",
      "Epoch [10800/20000], Loss: 0.0864\n",
      "Epoch [10900/20000], Loss: 0.0859\n",
      "Epoch [11000/20000], Loss: 0.0854\n",
      "Epoch [11100/20000], Loss: 0.0849\n",
      "Epoch [11200/20000], Loss: 0.0844\n",
      "Epoch [11300/20000], Loss: 0.0839\n",
      "Epoch [11400/20000], Loss: 0.0834\n",
      "Epoch [11500/20000], Loss: 0.0829\n",
      "Epoch [11600/20000], Loss: 0.0825\n",
      "Epoch [11700/20000], Loss: 0.0820\n",
      "Epoch [11800/20000], Loss: 0.0816\n",
      "Epoch [11900/20000], Loss: 0.0812\n",
      "Epoch [12000/20000], Loss: 0.0808\n",
      "Epoch [12100/20000], Loss: 0.0804\n",
      "Epoch [12200/20000], Loss: 0.0800\n",
      "Epoch [12300/20000], Loss: 0.0796\n",
      "Epoch [12400/20000], Loss: 0.0792\n",
      "Epoch [12500/20000], Loss: 0.0788\n",
      "Epoch [12600/20000], Loss: 0.0784\n",
      "Epoch [12700/20000], Loss: 0.0781\n",
      "Epoch [12800/20000], Loss: 0.0777\n",
      "Epoch [12900/20000], Loss: 0.0774\n",
      "Epoch [13000/20000], Loss: 0.0770\n",
      "Epoch [13100/20000], Loss: 0.0767\n",
      "Epoch [13200/20000], Loss: 0.0764\n",
      "Epoch [13300/20000], Loss: 0.0761\n",
      "Epoch [13400/20000], Loss: 0.0757\n",
      "Epoch [13500/20000], Loss: 0.0754\n",
      "Epoch [13600/20000], Loss: 0.0751\n",
      "Epoch [13700/20000], Loss: 0.0748\n",
      "Epoch [13800/20000], Loss: 0.0746\n",
      "Epoch [13900/20000], Loss: 0.0743\n",
      "Epoch [14000/20000], Loss: 0.0740\n",
      "Epoch [14100/20000], Loss: 0.0737\n",
      "Epoch [14200/20000], Loss: 0.0734\n",
      "Epoch [14300/20000], Loss: 0.0732\n",
      "Epoch [14400/20000], Loss: 0.0729\n",
      "Epoch [14500/20000], Loss: 0.0727\n",
      "Epoch [14600/20000], Loss: 0.0724\n",
      "Epoch [14700/20000], Loss: 0.0722\n",
      "Epoch [14800/20000], Loss: 0.0719\n",
      "Epoch [14900/20000], Loss: 0.0717\n",
      "Epoch [15000/20000], Loss: 0.0714\n",
      "Epoch [15100/20000], Loss: 0.0712\n",
      "Epoch [15200/20000], Loss: 0.0710\n",
      "Epoch [15300/20000], Loss: 0.0708\n",
      "Epoch [15400/20000], Loss: 0.0705\n",
      "Epoch [15500/20000], Loss: 0.0703\n",
      "Epoch [15600/20000], Loss: 0.0701\n",
      "Epoch [15700/20000], Loss: 0.0699\n",
      "Epoch [15800/20000], Loss: 0.0697\n",
      "Epoch [15900/20000], Loss: 0.0695\n",
      "Epoch [16000/20000], Loss: 0.0693\n",
      "Epoch [16100/20000], Loss: 0.0691\n",
      "Epoch [16200/20000], Loss: 0.0689\n",
      "Epoch [16300/20000], Loss: 0.0687\n",
      "Epoch [16400/20000], Loss: 0.0685\n",
      "Epoch [16500/20000], Loss: 0.0683\n",
      "Epoch [16600/20000], Loss: 0.0682\n",
      "Epoch [16700/20000], Loss: 0.0680\n",
      "Epoch [16800/20000], Loss: 0.0678\n",
      "Epoch [16900/20000], Loss: 0.0676\n",
      "Epoch [17000/20000], Loss: 0.0675\n",
      "Epoch [17100/20000], Loss: 0.0673\n",
      "Epoch [17200/20000], Loss: 0.0671\n",
      "Epoch [17300/20000], Loss: 0.0670\n",
      "Epoch [17400/20000], Loss: 0.0668\n",
      "Epoch [17500/20000], Loss: 0.0666\n",
      "Epoch [17600/20000], Loss: 0.0665\n",
      "Epoch [17700/20000], Loss: 0.0663\n",
      "Epoch [17800/20000], Loss: 0.0662\n",
      "Epoch [17900/20000], Loss: 0.0660\n",
      "Epoch [18000/20000], Loss: 0.0659\n",
      "Epoch [18100/20000], Loss: 0.0657\n",
      "Epoch [18200/20000], Loss: 0.0656\n",
      "Epoch [18300/20000], Loss: 0.0654\n",
      "Epoch [18400/20000], Loss: 0.0653\n",
      "Epoch [18500/20000], Loss: 0.0652\n",
      "Epoch [18600/20000], Loss: 0.0650\n",
      "Epoch [18700/20000], Loss: 0.0649\n",
      "Epoch [18800/20000], Loss: 0.0648\n",
      "Epoch [18900/20000], Loss: 0.0646\n",
      "Epoch [19000/20000], Loss: 0.0645\n",
      "Epoch [19100/20000], Loss: 0.0644\n",
      "Epoch [19200/20000], Loss: 0.0642\n",
      "Epoch [19300/20000], Loss: 0.0641\n",
      "Epoch [19400/20000], Loss: 0.0640\n",
      "Epoch [19500/20000], Loss: 0.0639\n",
      "Epoch [19600/20000], Loss: 0.0637\n",
      "Epoch [19700/20000], Loss: 0.0636\n",
      "Epoch [19800/20000], Loss: 0.0635\n",
      "Epoch [19900/20000], Loss: 0.0634\n",
      "Epoch [20000/20000], Loss: 0.0633\n",
      "Training time: 2.93 seconds\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/xnp5ZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABQ7ElEQVR4nO3deXgT1f4/8PckbdI1XeheWlrKUhbZpVZAUCsFuQguV0SUggrK4oZelauy+VVQULlXNvUKqFcF9Ce4sMgiXAWLrEX2HQqlC13TNWmT8/sj7UBoKaVNO0n6fj1PnjYzZ5LPdErz5sw5M5IQQoCIiIjISaiULoCIiIjIlhhuiIiIyKkw3BAREZFTYbghIiIip8JwQ0RERE6F4YaIiIicCsMNERERORWGGyIiInIqDDdERETkVBhuiOzImDFjEBUVVa9tZ8yYAUmSbFsQUQ2WL18OSZKwZ88epUshqhHDDVEdSJJUp8e2bduULlURY8aMgZeXl9JlOI2q8HC9x86dO5UukciuuShdAJEj+PLLL62ef/HFF9i0aVO15R06dGjQ+3z66acwm8312vaNN97Aa6+91qD3J/sya9YsREdHV1vepk0bBaohchwMN0R18Nhjj1k937lzJzZt2lRt+bVKSkrg4eFR5/dxdXWtV30A4OLiAhcX/pN2FMXFxfD09Ky1zeDBg9GrV68mqojIefC0FJGNDBgwAJ07d8bevXtxxx13wMPDA//85z8BAD/88AOGDBmCsLAwaLVaxMTE4K233oLJZLJ6jWvH3Jw7dw6SJGHevHn45JNPEBMTA61Wi1tvvRW7d++22ramMTeSJGHy5MlYs2YNOnfuDK1Wi06dOmHDhg3V6t+2bRt69eoFNzc3xMTE4OOPP7b5OJ5vv/0WPXv2hLu7OwICAvDYY48hLS3Nqk1GRgbGjh2Lli1bQqvVIjQ0FMOGDcO5c+fkNnv27EFiYiICAgLg7u6O6OhoPPHEE3WqYdGiRejUqRO0Wi3CwsIwadIk5Ofny+snT54MLy8vlJSUVNt25MiRCAkJsTpu69evR79+/eDp6Qlvb28MGTIEhw8fttqu6rTd6dOnce+998Lb2xujRo2qU721ufr348MPP0SrVq3g7u6O/v3749ChQ9Xa//rrr3Ktvr6+GDZsGI4ePVqtXVpaGp588kn59zU6OhoTJkyA0Wi0amcwGDBlyhQEBgbC09MT999/Py5fvmzVpiHHiqi++N88IhvKycnB4MGD8cgjj+Cxxx5DcHAwAMsYCi8vL0yZMgVeXl749ddfMW3aNOj1esydO/eGr/v111+jsLAQTz/9NCRJwnvvvYcHHngAZ86cuWFvz/bt2/H9999j4sSJ8Pb2xr///W88+OCDSE1NRYsWLQAA+/fvx6BBgxAaGoqZM2fCZDJh1qxZCAwMbPgPpdLy5csxduxY3HrrrZg9ezYyMzPxr3/9Czt27MD+/fvh6+sLAHjwwQdx+PBhPPvss4iKikJWVhY2bdqE1NRU+fnAgQMRGBiI1157Db6+vjh37hy+//77G9YwY8YMzJw5EwkJCZgwYQKOHz+OxYsXY/fu3dixYwdcXV0xYsQILFy4EGvXrsXf//53eduSkhL89NNPGDNmDNRqNQDL6cqkpCQkJibi3XffRUlJCRYvXoy+ffti//79VkG1oqICiYmJ6Nu3L+bNm1enHr2CggJkZ2dbLZMkST5uVb744gsUFhZi0qRJKCsrw7/+9S/cddddOHjwoPw7uHnzZgwePBitW7fGjBkzUFpaio8++gh9+vTBvn375FovXbqE3r17Iz8/H+PHj0dsbCzS0tLw3XffoaSkBBqNRn7fZ599Fn5+fpg+fTrOnTuH+fPnY/LkyVi5ciUANOhYETWIIKKbNmnSJHHtP5/+/fsLAGLJkiXV2peUlFRb9vTTTwsPDw9RVlYmL0tKShKtWrWSn589e1YAEC1atBC5ubny8h9++EEAED/99JO8bPr06dVqAiA0Go04deqUvOzAgQMCgPjoo4/kZUOHDhUeHh4iLS1NXnby5Enh4uJS7TVrkpSUJDw9Pa+73mg0iqCgING5c2dRWloqL//5558FADFt2jQhhBB5eXkCgJg7d+51X2v16tUCgNi9e/cN67paVlaW0Gg0YuDAgcJkMsnLFyxYIACIpUuXCiGEMJvNIjw8XDz44INW269atUoAEL/99psQQojCwkLh6+srxo0bZ9UuIyND+Pj4WC1PSkoSAMRrr71Wp1qXLVsmANT40Gq1cruq3w93d3dx8eJFefmff/4pAIgXX3xRXtatWzcRFBQkcnJy5GUHDhwQKpVKjB49Wl42evRooVKpavz5ms1mq/oSEhLkZUII8eKLLwq1Wi3y8/OFEPU/VkQNxdNSRDak1WoxduzYasvd3d3l7wsLC5GdnY1+/fqhpKQEx44du+HrjhgxAn5+fvLzfv36AQDOnDlzw20TEhIQExMjP+/SpQt0Op28rclkwubNmzF8+HCEhYXJ7dq0aYPBgwff8PXrYs+ePcjKysLEiRPh5uYmLx8yZAhiY2Oxdu1aAJafk0ajwbZt25CXl1fja1X18Pz8888oLy+vcw2bN2+G0WjECy+8AJXqyp++cePGQafTyTVIkoS///3vWLduHYqKiuR2K1euRHh4OPr27QsA2LRpE/Lz8zFy5EhkZ2fLD7Vajbi4OGzdurVaDRMmTKhzvQCwcOFCbNq0yeqxfv36au2GDx+O8PBw+Xnv3r0RFxeHdevWAQDS09ORkpKCMWPGwN/fX27XpUsX3HPPPXI7s9mMNWvWYOjQoTWO9bn2FOX48eOtlvXr1w8mkwnnz58HUP9jRdRQDDdENhQeHm7VbV/l8OHDuP/+++Hj4wOdTofAwEB5MHJBQcENXzcyMtLqeVXQuV4AqG3bqu2rts3KykJpaWmNM3BsNSun6sOuffv21dbFxsbK67VaLd59912sX78ewcHBuOOOO/Dee+8hIyNDbt+/f388+OCDmDlzJgICAjBs2DAsW7YMBoOhXjVoNBq0bt1aXg9YwmRpaSl+/PFHAEBRURHWrVuHv//97/KH+cmTJwEAd911FwIDA60eGzduRFZWltX7uLi4oGXLljf+YV2ld+/eSEhIsHrceeed1dq1bdu22rJ27drJ45Rq+/l36NAB2dnZKC4uxuXLl6HX69G5c+c61Xej38v6HiuihmK4IbKhq3toquTn56N///44cOAAZs2ahZ9++gmbNm3Cu+++CwB1mvpdNcbjWkKIRt1WCS+88AJOnDiB2bNnw83NDW+++SY6dOiA/fv3A7D0Hnz33XdITk7G5MmTkZaWhieeeAI9e/a06mlpiNtuuw1RUVFYtWoVAOCnn35CaWkpRowYIbepOm5ffvlltd6VTZs24YcffrB6Ta1Wa9Vj5Axu9LvVFMeKqCbO9S+NyA5t27YNOTk5WL58OZ5//nn87W9/Q0JCgtVpJiUFBQXBzc0Np06dqraupmX10apVKwDA8ePHq607fvy4vL5KTEwMXnrpJWzcuBGHDh2C0WjE+++/b9Xmtttuw9tvv409e/bgq6++wuHDh7FixYqbrsFoNOLs2bPVanj44YexYcMG6PV6rFy5ElFRUbjtttusagQsP79re1cSEhIwYMCAG/xUbKeqF+lqJ06ckAcJ1/bzP3bsGAICAuDp6YnAwEDodLoaZ1o1xM0eK6KGYrghamRV/7u9uqfEaDRi0aJFSpVkRa1WIyEhAWvWrMGlS5fk5adOnapxfEd99OrVC0FBQViyZInVKYn169fj6NGjGDJkCADLjKSysjKrbWNiYuDt7S1vl5eXV63XqVu3bgBQ6+mOhIQEaDQa/Pvf/7ba/rPPPkNBQYFcQ5URI0bAYDDg888/x4YNG/Dwww9brU9MTIROp8M777xT43iSa6dEN6Y1a9ZYTanftWsX/vzzT3nMVGhoKLp164bPP//catr7oUOHsHHjRtx7770AAJVKheHDh+Onn36q8dYKN9vbV99jRdRQnApO1Mhuv/12+Pn5ISkpCc899xwkScKXX35pV6eFZsyYgY0bN6JPnz6YMGECTCYTFixYgM6dOyMlJaVOr1FeXo7/+7//q7bc398fEydOxLvvvouxY8eif//+GDlypDwVPCoqCi+++CIAS2/D3XffjYcffhgdO3aEi4sLVq9ejczMTDzyyCMAgM8//xyLFi3C/fffj5iYGBQWFuLTTz+FTqeTP6RrEhgYiKlTp2LmzJkYNGgQ7rvvPhw/fhyLFi3CrbfeWu2CjD169ECbNm3w+uuvw2AwWJ2SAgCdTofFixfj8ccfR48ePfDII48gMDAQqampWLt2Lfr06YMFCxbU6Wd3PevXr69xwPntt9+O1q1by8/btGmDvn37YsKECTAYDJg/fz5atGiBV155RW4zd+5cDB48GPHx8XjyySflqeA+Pj6YMWOG3O6dd97Bxo0b0b9/f4wfPx4dOnRAeno6vv32W2zfvl0eJFwX9T1WRA2m2DwtIgd2vangnTp1qrH9jh07xG233Sbc3d1FWFiYeOWVV8Qvv/wiAIitW7fK7a43FbymqdEAxPTp0+Xn15sKPmnSpGrbtmrVSiQlJVkt27Jli+jevbvQaDQiJiZG/Oc//xEvvfSScHNzu85P4Yqqqc41PWJiYuR2K1euFN27dxdarVb4+/uLUaNGWU1hzs7OFpMmTRKxsbHC09NT+Pj4iLi4OLFq1Sq5zb59+8TIkSNFZGSk0Gq1IigoSPztb38Te/bsuWGdQlimfsfGxgpXV1cRHBwsJkyYIPLy8mps+/rrrwsAok2bNtd9va1bt4rExETh4+Mj3NzcRExMjBgzZoxVPTeaKn+t2qaCAxDLli0TQlj/frz//vsiIiJCaLVa0a9fP3HgwIFqr7t582bRp08f4e7uLnQ6nRg6dKg4cuRItXbnz58Xo0ePFoGBgUKr1YrWrVuLSZMmCYPBYFXftVO8t27davU73dBjRVRfkhB29N9HIrIrw4cPx+HDh2sc00HKO3fuHKKjozF37ly8/PLLSpdDZDc45oaIAAClpaVWz0+ePIl169Y16cBYIiJb4JgbIgIAtG7dGmPGjJGv+bJ48WJoNBqrcRtERI6A4YaIAACDBg3CN998g4yMDGi1WsTHx+Odd96p8QJxRET2jGNuiIiIyKlwzA0RERE5FYYbIiIicirNbsyN2WzGpUuX4O3tXe0Ot0RERGSfhBAoLCxEWFjYDe/T1uzCzaVLlxAREaF0GURERFQPFy5cQMuWLWtt0+zCjbe3NwDLD0en0ylcDREREdWFXq9HRESE/DleG0XDzW+//Ya5c+di7969SE9Px+rVqzF8+PDrtv/++++xePFipKSkwGAwoFOnTpgxYwYSExPr/J5Vp6J0Oh3DDRERkYOpy5ASRQcUFxcXo2vXrli4cGGd2v/222+45557sG7dOuzduxd33nknhg4div379zdypUREROQo7OY6N5Ik3bDnpiadOnXCiBEjMG3atDq11+v18PHxQUFBAXtuiIiIHMTNfH479Jgbs9mMwsJC+Pv7X7eNwWCAwWCQn+v1+qYojYiIiBTi0Ne5mTdvHoqKivDwww9ft83s2bPh4+MjPzhTioiIyLk5bLj5+uuvMXPmTKxatQpBQUHXbTd16lQUFBTIjwsXLjRhlURERNTUHPK01IoVK/DUU0/h22+/RUJCQq1ttVottFptE1VGRERESnO4nptvvvkGY8eOxTfffIMhQ4YoXQ4RERHZGUV7boqKinDq1Cn5+dmzZ5GSkgJ/f39ERkZi6tSpSEtLwxdffAHAcioqKSkJ//rXvxAXF4eMjAwAgLu7O3x8fBTZByIiIrIvivbc7NmzB927d0f37t0BAFOmTEH37t3lad3p6elITU2V23/yySeoqKjApEmTEBoaKj+ef/55ReonIiIi+2M317lpKrzODRERkeO5mc9vhxtzQ0RERFQbhhsiIiJyKgw3NiKEwOVCA85mFytdChERUbPGcGMj/ztxGbe+vRkT/rtX6VKIiIiaNYYbG2np5w4ASMsrVbgSIiKi5o3hxkbCfC3hptBQgYLScoWrISIiar4YbmzEQ+MCPw9XAMClfPbeEBERKYXhxobCeWqKiIhIcQw3NhReeWoqjT03REREimG4saFwXw8APC1FRESkJIYbGwrzdQMAXGS4ISIiUgzDjQ1F+lt6bs7n8EJ+RERESmG4saGYIC8AwOmsYpjNzep+pERERHaD4caGIv094KKSUFpuQoa+TOlyiIiImiWGGxtyVavQqoXl1NTpy0UKV0NERNQ8MdzYWExg1akphhsiIiIlMNzYWNW4m1PsuSEiIlIEw42NtQ/2BgAcTS9UuBIiIqLmieHGxjqH+wAAjlzSw8QZU0RERE2O4cbGWgd4wlOjRmm5iYOKiYiIFMBwY2MqlYROYZbem4MXCxSuhoiIqPlhuGkEVaemDqYx3BARETU1hptG0DXCEm72ns9TuBIiIqLmh+GmEcRFtwAAHL5UgILScoWrISIial4YbhpBiI8bogM8YRbA7rO5SpdDRETUrDDcNJLbWlt6b3aeyVG4EiIiouaF4aaR3NbaHwDwx2mGGyIioqbEcNNI4mMsPTdHM/TILTYqXA0REVHzwXDTSIK83dA+2BtCAMnsvSEiImoyDDeNqE+bAADA9lPZCldCRETUfDDcNKK+bS2npnYw3BARETUZhptG1Du6BVxUElJzS5CaU6J0OURERM0Cw00j8tK6oHukLwBgx2n23hARETUFhptGxnE3RERETYvhppH1rQw3f5zKhtksFK6GiIjI+THcNLKuEb7w1KiRV1KOoxl6pcshIiJyegw3jcxVrZJvxcBZU0RERI2P4aYJXBl3w4v5ERERNTaGmybQt60l3Ow6mwNDhUnhaoiIiJwbw00TaBvkhUBvLcrKzdh3Pl/pcoiIiJwaw00TkCQJfWI47oaIiKgpMNw0EV7vhoiIqGkw3DSRqnDz18V8FJSWK1wNERGR82K4aSJhvu5oHegJswB2n81VuhwiIiKnxXDThHpE+gEA/korULgSIiIi58Vw04S6tPQBABy8mK9sIURERE6M4aYJdWnpCwD462IBhOB9poiIiBoDw00Tig3xhotKQk6xEZcKypQuh4iIyCkx3DQhN1c12od4A+CpKSIiosbCcNPEqsbdHLjIQcVERESNQdFw89tvv2Ho0KEICwuDJElYs2bNDbfZtm0bevToAa1WizZt2mD58uWNXqctdQyzhJtj6XqFKyEiInJOioab4uJidO3aFQsXLqxT+7Nnz2LIkCG48847kZKSghdeeAFPPfUUfvnll0au1HZiK09LncgsUrgSIiIi5+Si5JsPHjwYgwcPrnP7JUuWIDo6Gu+//z4AoEOHDti+fTs+/PBDJCYmNlaZNtUuyBJu0vJLoS8rh87NVeGKiIiInItDjblJTk5GQkKC1bLExEQkJydfdxuDwQC9Xm/1UJKPhytCdG4AgJOZhYrWQkRE5IwcKtxkZGQgODjYallwcDD0ej1KS0tr3Gb27Nnw8fGRHxEREU1Raq3aVZ6aOp7BU1NERES25lDhpj6mTp2KgoIC+XHhwgWlS7pq3A17boiIiGxN0TE3NyskJASZmZlWyzIzM6HT6eDu7l7jNlqtFlqttinKq7N2wZZwcyyDM6aIiIhszaF6buLj47FlyxarZZs2bUJ8fLxCFdVP+2DOmCIiImosioaboqIipKSkICUlBYBlqndKSgpSU1MBWE4pjR49Wm7/zDPP4MyZM3jllVdw7NgxLFq0CKtWrcKLL76oRPn11jrQEwCQW2xEfolR4WqIiIici6LhZs+ePejevTu6d+8OAJgyZQq6d++OadOmAQDS09PloAMA0dHRWLt2LTZt2oSuXbvi/fffx3/+8x+HmQZexVPrgmCd5VTZ2exihashIiJyLoqOuRkwYECtd8eu6erDAwYMwP79+xuxqqYRHeCJTL0B53KK0T3ST+lyiIiInIZDjblxJtEBXgCAs5fZc0NERGRLDDcKaR1gGXdzhqeliIiIbIrhRiFRleGGY26IiIhsi+FGIdGV4eZcdnGt446IiIjo5jDcKCTS3wMqCSg2mnC50KB0OURERE6D4UYhGhcVIvw9AACnOaiYiIjIZhhuFNSqheXU1IXcEoUrISIich4MNwqK8LPcD+tCHsMNERGRrTDcKKiln+W01MW8UoUrISIich4MNwqK8K/sueFpKSIiIpthuFFQRGXPDU9LERER2Q7DjYKqZktl6g0oKzcpXA0REZFzYLhRkJ+HKzw0agDApXyOuyEiIrIFhhsFSZJ01akphhsiIiJbYLhRGAcVExER2RbDjcI4HZyIiMi2GG4U1pIX8iMiIrIphhuFVc2YusjTUkRERDbBcKOwcF9Lz00aZ0sRERHZBMONwqrCTXaREYYKXuuGiIiooRhuFObr4Qo3V8thyCgoU7gaIiIix8dwozBJkhDmY+m9uZTPcENERNRQDDd2INTXDQCvUkxERGQLDDd2oKrnJr2A4YaIiKihGG7sQGjloOJLHHNDRETUYAw3diDMx3JaKp2npYiIiBqM4cYOyD03HFBMRETUYAw3diC8akAxx9wQERE1GMONHQitHFBcWFaBIkOFwtUQERE5NoYbO+CpdYHOzQUAx90QERE1FMONnQjjPaaIiIhsguHGTlSFm3ROByciImoQhhs7Ecrp4ERERDbBcGMnrpyWYs8NERFRQzDc2Am554bTwYmIiBqE4cZOhPpwzA0REZEtMNzYibCr7gwuhFC4GiIiIsfFcGMngnWWcGOoMCO/pFzhaoiIiBwXw42dcHNVo4WnBgBvw0BERNQQDDd2JLTy1FQGx90QERHVG8ONHakaVHyJ4YaIiKjeGG7sSNV08AyeliIiIqo3hhs7Ik8H54X8iIiI6o3hxo5cuZAfww0REVF9MdzYEV6lmIiIqOEYbuzI1Vcp5oX8iIiI6ofhxo4E+2gBWC7kl8cL+REREdULw40d0bqoEeBVeSG/fJ6aIiIiqg+GGztTdWqKF/IjIiKqH4YbOxPCQcVEREQNoni4WbhwIaKiouDm5oa4uDjs2rWr1vbz589H+/bt4e7ujoiICLz44osoK3OeXo4wTgcnIiJqEEXDzcqVKzFlyhRMnz4d+/btQ9euXZGYmIisrKwa23/99dd47bXXMH36dBw9ehSfffYZVq5ciX/+859NXHnjCblqxhQRERHdPEXDzQcffIBx48Zh7Nix6NixI5YsWQIPDw8sXbq0xvZ//PEH+vTpg0cffRRRUVEYOHAgRo4cecPeHkcS5svTUkRERA2hWLgxGo3Yu3cvEhISrhSjUiEhIQHJyck1bnP77bdj7969cpg5c+YM1q1bh3vvvfe672MwGKDX660e9iyUPTdEREQN4qLUG2dnZ8NkMiE4ONhqeXBwMI4dO1bjNo8++iiys7PRt29fCCFQUVGBZ555ptbTUrNnz8bMmTNtWntjuvoWDEIISJKkcEVERESORfEBxTdj27ZteOedd7Bo0SLs27cP33//PdauXYu33nrruttMnToVBQUF8uPChQtNWPHNC9ZZwo2xwozcYqPC1RARETkexXpuAgICoFarkZmZabU8MzMTISEhNW7z5ptv4vHHH8dTTz0FALjllltQXFyM8ePH4/XXX4dKVT2rabVaaLVa2+9AI9G4qBDgpUV2kQHpBWVo4eU4tRMREdkDxXpuNBoNevbsiS1btsjLzGYztmzZgvj4+Bq3KSkpqRZg1Go1ADjVvZiuDCrmuBsiIqKbpVjPDQBMmTIFSUlJ6NWrF3r37o358+ejuLgYY8eOBQCMHj0a4eHhmD17NgBg6NCh+OCDD9C9e3fExcXh1KlTePPNNzF06FA55DiDEJ0b/kIBZ0wRERHVg6LhZsSIEbh8+TKmTZuGjIwMdOvWDRs2bJAHGaemplr11LzxxhuQJAlvvPEG0tLSEBgYiKFDh+Ltt99WahcaRZgvZ0wRERHVlySc6XxOHej1evj4+KCgoAA6nU7pcmq05H+nMWf9MQzvFob5j3RXuhwiIiLF3cznt0PNlmouQnkLBiIionpjuLFDvJAfERFR/THc2KGqnpuMygv5ERERUd0x3NihYJ0bJAkwmszI4YX8iIiIbgrDjR2qupAfAKTn89QUERHRzWC4sVNhPrw7OBERUX0w3NipEM6YIiIiqheGGzvFGVNERET1w3Bjp0J5WoqIiKheGG7sVChvwUBERFQvDDd2ij03RERE9cNwY6eqwk1mgQFmMy/kR0REVFcMN3aKF/IjIiKqH4YbO+WqViGw8kJ+GRx3Q0REVGcMN3as6tTUJY67ISIiqjOGGzsmX+smn+GGiIiorhhu7Jh8lWI9T0sRERHVFcONHQvzrQw3vHkmERFRnTHc2LGQytNSHFBMRERUdww3diyMA4qJiIhuGsONHau6BUOmvowX8iMiIqojhhs7FuSthSQB5SaB7GKD0uUQERE5BIYbO+aqViHImxfyIyIiuhkMN3aualDxJc6YIiIiqhOGGzvXsnLczcW8EoUrISIicgwMN3Yuwt8DAHAhl+GGiIioLhhu7FyrFpZwc57hhoiIqE4YbuxcZGXPTSrDDRERUZ0w3Ni5qnBzMbcUJl7rhoiI6IYYbuxcqI8bXFQSjCYzMnkDTSIiohtiuLFzLmoVWvpZZkydz+GpKSIiohthuHEAnDFFRERUdww3DuDKjKlihSshIiKyfww3DuDKjCneHZyIiOhGGG4cQKS/JwAgNYc9N0RERDfCcOMAqk5L8Vo3REREN8Zw4wCqBhTnlZSjoLRc4WqIiIjsG8ONA/DSuiDQWwsAOJfNU1NERES1qVe4uXDhAi5evCg/37VrF1544QV88sknNiuMrMUEWsbdnL5cpHAlRERE9q1e4ebRRx/F1q1bAQAZGRm45557sGvXLrz++uuYNWuWTQski9aBXgCAM5fZc0NERFSbeoWbQ4cOoXfv3gCAVatWoXPnzvjjjz/w1VdfYfny5basjyrFVIYb9twQERHVrl7hpry8HFqtZQzI5s2bcd999wEAYmNjkZ6ebrvqSMbTUkRERHVTr3DTqVMnLFmyBL///js2bdqEQYMGAQAuXbqEFi1a2LRAsqjquTmXXcK7gxMREdWiXuHm3Xffxccff4wBAwZg5MiR6Nq1KwDgxx9/lE9XkW2F+bpD66KC0WTGxTxe74aIiOh6XOqz0YABA5CdnQ29Xg8/Pz95+fjx4+Hh4WGz4ugKtUpCdIAnjmUU4vTlIrRq4al0SURERHapXj03paWlMBgMcrA5f/485s+fj+PHjyMoKMimBdIV8qDiLM6YIiIiup56hZthw4bhiy++AADk5+cjLi4O77//PoYPH47FixfbtEC6ompQ8ZlsDiomIiK6nnqFm3379qFfv34AgO+++w7BwcE4f/48vvjiC/z73/+2aYF0RUyQpefmVBbDDRER0fXUK9yUlJTA29sbALBx40Y88MADUKlUuO2223D+/HmbFkhXtA2y/MyPZRRCCM6YIiIiqkm9wk2bNm2wZs0aXLhwAb/88gsGDhwIAMjKyoJOp7NpgXRFTJAnXFQSCssqkF5QpnQ5REREdqle4WbatGl4+eWXERUVhd69eyM+Ph6ApRene/fuN/VaCxcuRFRUFNzc3BAXF4ddu3bV2j4/Px+TJk1CaGgotFot2rVrh3Xr1tVnNxyO1kUtDyo+lqFXuBoiIiL7VK+p4A899BD69u2L9PR0+Ro3AHD33Xfj/vvvr/PrrFy5ElOmTMGSJUsQFxeH+fPnIzEx8bqzroxGI+655x4EBQXhu+++Q3h4OM6fPw9fX9/67IZDig31xvHMQhxNL8RdscFKl0NERGR36hVuACAkJAQhISHy3cFbtmx50xfw++CDDzBu3DiMHTsWALBkyRKsXbsWS5cuxWuvvVat/dKlS5Gbm4s//vgDrq6uAICoqKj67oJDig3R4QdcwrGMQqVLISIiskv1Oi1lNpsxa9Ys+Pj4oFWrVmjVqhV8fX3x1ltvwWw21+k1jEYj9u7di4SEhCvFqFRISEhAcnJyjdv8+OOPiI+Px6RJkxAcHIzOnTvjnXfegclkuu77GAwG6PV6q4cjiw2tHFSc7tj7QURE1Fjq1XPz+uuv47PPPsOcOXPQp08fAMD27dsxY8YMlJWV4e23377ha2RnZ8NkMiE42PrUSnBwMI4dO1bjNmfOnMGvv/6KUaNGYd26dTh16hQmTpyI8vJyTJ8+vcZtZs+ejZkzZ97kHtqvDiGWAdtnsotRVm6Cm6ta4YqIiIjsS73Czeeff47//Oc/8t3AAaBLly4IDw/HxIkT6xRu6sNsNiMoKAiffPIJ1Go1evbsibS0NMydO/e64Wbq1KmYMmWK/Fyv1yMiIqJR6msKwTotfD1ckV9SjlNZRegc7qN0SURERHalXuEmNzcXsbGx1ZbHxsYiNze3Tq8REBAAtVqNzMxMq+WZmZkICQmpcZvQ0FC4urpCrb7SW9GhQwdkZGTAaDRCo9FU20ar1UKr1dapJkcgSRJiQ7yx80wujqbrGW6IiIiuUa8xN127dsWCBQuqLV+wYAG6dOlSp9fQaDTo2bMntmzZIi8zm83YsmWLPLX8Wn369MGpU6esxvWcOHECoaGhNQYbZ9UpzBJoDqYVKFwJERGR/alXz817772HIUOGYPPmzXIQSU5OxoULF27qmjNTpkxBUlISevXqhd69e2P+/PkoLi6WZ0+NHj0a4eHhmD17NgBgwoQJWLBgAZ5//nk8++yzOHnyJN555x0899xz9dkNh9WlpSXcHLjIcENERHStevXc9O/fHydOnMD999+P/Px85Ofn44EHHsDhw4fx5Zdf1vl1RowYgXnz5mHatGno1q0bUlJSsGHDBnmQcWpqKtLT0+X2ERER+OWXX7B792506dIFzz33HJ5//vkap407s64tfQEARy/pYayo2+w0IiKi5kISNrxJ0YEDB9CjR49ap2YrTa/Xw8fHBwUFBQ57qwghBLrN2oSC0nL8NLkvbmnJcTdEROTcbubzu149N6QsSZKuOjWVr2wxREREdobhxkFVnZr6i+GGiIjICsONg5J7bi5wUDEREdHVbmq21AMPPFDr+vz8/IbUQjeha4QvAOBkViGKDRXw1Nb7NmFERERO5aY+EX18ah+46uPjg9GjRzeoIKqbYJ0bwn3dkZZfiv2p+ejbNkDpkoiIiOzCTYWbZcuWNVYdVA+3RvkhLaUUu87mMNwQERFV4pgbB3ZrtD8AYNe5ut3ygoiIqDlguHFgvaMs4WZ/aj4v5kdERFSJ4caBtQnygr+nBoYKM+8zRUREVInhxoFJkoRerfwAALt5aoqIiAgAw43D61057ubPMzkKV0JERGQfGG4cXHxMCwDAn2dzOe6GiIgIDDcOr0OIDgFeGpQYTdh7Pk/pcoiIiBTHcOPgVCoJ/doGAgB+P3lZ4WqIiIiUx3DjBPpVXsDvN4YbIiIihhtnUHV14kNpeuQUGRSuhoiISFkMN04gyNsNHUJ1AIDtp7IVroaIiEhZDDdOon87y7ibLUezFK6EiIhIWQw3TuKejsEAgK3HsjglnIiImjWGGyfRPcIXAV5aFBoqsJMX9CMiomaM4cZJqFSS3Huz8UiGwtUQEREph+HGiQzsZAk3m45kwmwWCldDRESkDIYbJ3J7TAt4aV2QqTcg5WK+0uUQEREpguHGiWhd1LgrNggA8NOBSwpXQ0REpAyGGyczrFsYAOCnA+moMHHWFBERNT8MN07mjnaB8PNwRXaRAX+c5qwpIiJqfhhunIyrWoUhXUIBAGtS0hSuhoiIqOkx3Dih4d3CAQC/HMpAqdGkcDVERERNi+HGCfWI9EO4rzuKjSZe84aIiJodhhsnpFJJeKhnSwDAN7tSFa6GiIioaTHcOKmHb42ASgJ2nsnFmctFSpdDRETUZBhunFS4rzsGtLdc82bl7gsKV0NERNR0GG6c2MjekQCAb/dehKGCA4uJiKh5YLhxYne2D0SwTovcYiN+OZypdDlERERNguHGibmoVXjkVkvvzbIdZxWuhoiIqGkw3Di5x25rBY1ahf2p+dh7Pk/pcoiIiBodw42TC/TWyvebWrqdvTdEROT8GG6agSf7RQMA1h9Kx4XcEoWrISIialwMN81AbIgOfdsEwCyA5X+cU7ocIiKiRsVw00xU9d58sysVucVGhashIiJqPAw3zcSAdoHoFKZDidHEsTdEROTUGG6aCUmS8OxdbQAAn/9xDgUl5QpXRERE1DgYbpqRgR1D0D7YG4WGCiz7g703RETknBhumhGVSsLkyt6bpdvPorCMvTdEROR8GG6amXtvCUVMoCf0ZRVYuv2c0uUQERHZHMNNM6NWSXghoR0A4JPfTiOnyKBwRURERLbFcNMMDbklFJ3DdSg2mrBg6ymlyyEiIrIphptmSKWS8OqgWADAVztTedViIiJyKgw3zVS/toHo06YFjCYzPtx8QulyiIiIbIbhphl7JdHSe7N6fxqOZegVroaIiMg27CLcLFy4EFFRUXBzc0NcXBx27dpVp+1WrFgBSZIwfPjwxi3QSXWN8MWQW0IhBPDWz0cghFC6JCIiogZTPNysXLkSU6ZMwfTp07Fv3z507doViYmJyMrKqnW7c+fO4eWXX0a/fv2aqFLn9OqgWGhcVNhxKgebjmQqXQ4REVGDKR5uPvjgA4wbNw5jx45Fx44dsWTJEnh4eGDp0qXX3cZkMmHUqFGYOXMmWrdu3YTVOp/IFh4YV3lTzf9bexSGCpPCFRERETWMouHGaDRi7969SEhIkJepVCokJCQgOTn5utvNmjULQUFBePLJJ2/4HgaDAXq93upB1iYOaINgnRapuSW8sB8RETk8RcNNdnY2TCYTgoODrZYHBwcjIyOjxm22b9+Ozz77DJ9++mmd3mP27Nnw8fGRHxEREQ2u29l4al3w2mDL4OIFv55Elr5M4YqIiIjqT/HTUjejsLAQjz/+OD799FMEBATUaZupU6eioKBAfly4cKGRq3RMw7qGo3ukL4qNJvzf2qNKl0NERFRvLkq+eUBAANRqNTIzrQeyZmZmIiQkpFr706dP49y5cxg6dKi8zGw2AwBcXFxw/PhxxMTEWG2j1Wqh1WoboXrnolJJeGtYZ9y3YDt+PHAJD/VsiTvaBSpdFhER0U1TtOdGo9GgZ8+e2LJli7zMbDZjy5YtiI+Pr9Y+NjYWBw8eREpKivy47777cOeddyIlJYWnnBqoc7gPkm6PAgC8+cMhlJVzcDERETkeRXtuAGDKlClISkpCr1690Lt3b8yfPx/FxcUYO3YsAGD06NEIDw/H7Nmz4ebmhs6dO1tt7+vrCwDVllP9vDSwPdYfzMD5nBIs+PUUXk5sr3RJREREN0XxcDNixAhcvnwZ06ZNQ0ZGBrp164YNGzbIg4xTU1OhUjnU0CCH5qV1wYz7OuGZ/+7Fx7+dxrBuYWgb7K10WURERHUmiWZ2WVq9Xg8fHx8UFBRAp9MpXY5dEkJg3Bd7sPloFnpH+WPF+NugUklKl0VERM3YzXx+s0uEqpEkCTOHdYaHRo1d53Lx9a5UpUsiIiKqM4YbqlG4rzv+UTneZva6o7iQW6JwRURERHXDcEPXlRQfhd5R/ig2mvDq//uLN9YkIiKHwHBD16VSSXjvoS5wc1Xhj9M5+OpPnp4iIiL7x3BDtYoK8MSrgyy3ZuDpKSIicgQMN3RDPD1FRESOhOGGbuja01Nf7jyvdElERETXxXBDdRIV4InXKk9Pvb32KE5mFipcERERUc0YbqjOkm6Pwh3tAmGoMOP5FSkwVPDeU0REZH8YbqjOJEnCvIe6wN9TgyPpenyw8YTSJREREVXDcEM3JUjnhjkP3AIA+OT3M/jjdLbCFREREVljuKGbNrBTCEb2joQQwJSVB5BfYlS6JCIiIhnDDdXLm3/rgNYBnsjQl+Gfqw9yejgREdkNhhuqFw+NC+Y/0g0uKgnrDmbw5ppERGQ3GG6o3rq09JWvXjzzpyM4fKlA4YqIiIgYbqiBnuoXjYQOQTBWmDH56/0oLCtXuiQiImrmGG6oQSRJwry/d0W4rzvOZhdj6vccf0NERMpiuKEG8/XQ4N8ju8NFJeHnv9I5/oaIiBTFcEM20bOVn9X4m0NpHH9DRETKYLghm7l6/M3Er/bx+jdERKQIhhuymarxNxH+7kjNLcFzK1JgMnP8DRERNS2GG7IpXw8NPn6sF9xcVfjtxGW8v/G40iUREVEzw3BDNtcxTId3H+wCAFi07TTWHUxXuCIiImpOGG6oUQzrFo5x/aIBAC9/ewAnMgsVroiIiJoLhhtqNK8OisXtMS1QYjRh/Bd7UFDCC/wREVHjY7ihRuOiVmHBoz0Q7uuOczklmPj1XpSbzEqXRURETo7hhhqVv6cGn47uBQ+NGjtO5WDaD4d4BWMiImpUDDfU6DqG6fDvR7pDkoBvdl3Af34/q3RJRETkxBhuqEkkdAzG6/d2AAC8s/4oNh7OULgiIiJyVgw31GSe7BuNUXGREAJ4fkUKb9FARESNguGGmowkSZhxXyf0axuA0nITnvx8N9LyS5Uui4iInAzDDTUp18oZVG2DvJCpN2D0Z38ir5j3oCIiItthuKEm5+Puis+f6I1QHzecvlyMJz7fjRJjhdJlERGRk2C4IUWE+brj8yd6w8fdFftT8zH56/28Bg4REdkEww0ppl2wN5aO6QWtiwq/HsvC1O8P8ho4RETUYAw3pKierfyx8NEeUKskfLf3IuasP8aAQ0REDcJwQ4pL6BiM2fffAgD4+Lcz+NeWkwpXREREjozhhuzCw7dG4M2/dQQAzN98Ekv+d1rhioiIyFEx3JDdeLJvNP6R2B4AMGf9MSzbwds0EBHRzWO4Ibsy6c42eO7utgCAmT8dwdd/pipcERERORqGG7I7Lya0xdN3tAYAvL7mIFbuZsAhIqK6Y7ghuyNJEl4bHIsxt0dBCODV/3eQp6iIiKjOGG7ILkmShOlDO2Jcv2gAllNUC349yWniRER0Qww3ZLckScI/7+2AFxIsY3DmbTyBdzccZ8AhIqJaMdyQXZMkCS8ktMMbQzoAAJb87zRe/vYvGCt4qwYiIqoZww05hKf6tcacB26BWiXh/+27iLHLd0FfVq50WUREZIcYbshhPNI7Ev9J6gUPjRo7TuXg74uTcSm/VOmyiIjIzjDckEO5s30QVj0dj0BvLY5nFuL+RTuQciFf6bKIiMiOMNyQw+kc7oPVE29H2yAvZOoNeHhJMlbtvqB0WUREZCcYbsghtfTzwPcTb8fAjsEwmsx45f/9hTfWHORAYyIiso9ws3DhQkRFRcHNzQ1xcXHYtWvXddt++umn6NevH/z8/ODn54eEhIRa25Pz8nZzxZLHeuKle9pBkoD/7kzFyE93Io3jcIiImjXFw83KlSsxZcoUTJ8+Hfv27UPXrl2RmJiIrKysGttv27YNI0eOxNatW5GcnIyIiAgMHDgQaWlpTVw52QOVSsKzd7fFZ0m94O3mgr3n8zB4/m9YfzBd6dKIiEghklD4imhxcXG49dZbsWDBAgCA2WxGREQEnn32Wbz22ms33N5kMsHPzw8LFizA6NGjb9her9fDx8cHBQUF0Ol0Da6f7EdqTgmeW7FfHmD8aFwkpv2tI9xc1coWRkREDXYzn9+K9twYjUbs3bsXCQkJ8jKVSoWEhAQkJyfX6TVKSkpQXl4Of3//GtcbDAbo9XqrBzmnyBYe+PaZeDzTPwYA8PWfqRj60XYcvFigcGVERNSUFA032dnZMJlMCA4OtloeHByMjIyMOr3Gq6++irCwMKuAdLXZs2fDx8dHfkRERDS4brJfrmoVXhsciy+f7I1Aby1OZhVh+KIdmPvLMRgqTEqXR0RETUDxMTcNMWfOHKxYsQKrV6+Gm5tbjW2mTp2KgoIC+XHhAqcMNwf92gZiw/P9MLRrGExmgYVbT+Nv/97Oa+IQETUDioabgIAAqNVqZGZmWi3PzMxESEhIrdvOmzcPc+bMwcaNG9GlS5frttNqtdDpdFYPah5aeGnx0cjuWPJYDwR4aXAyqwgPLNqBmT8d5q0biIicmKLhRqPRoGfPntiyZYu8zGw2Y8uWLYiPj7/udu+99x7eeustbNiwAb169WqKUsmBDeocik0v9sfwbmEwC2DZjnO4a97/sHr/Rd5hnIjICSl+WmrKlCn49NNP8fnnn+Po0aOYMGECiouLMXbsWADA6NGjMXXqVLn9u+++izfffBNLly5FVFQUMjIykJGRgaKiIqV2gRyAn6cG8x/pji+e6I3WAZ7ILjLgxZUHMOLjnTiWwUHmRETORPFwM2LECMybNw/Tpk1Dt27dkJKSgg0bNsiDjFNTU5GefuWaJYsXL4bRaMRDDz2E0NBQ+TFv3jyldoEcyB3tArH+hX74R2J7uLmqsOtcLu791+949bu/kFFQpnR5RERkA4pf56ap8To3VCUtvxRvrz2CdQctM/PcXFV4sm80nu4fA52bq8LVERHR1W7m85vhhpq9vefzMHvdUew5nwcA8PfUYOKAGDwaFwkPjYvC1REREcBwUyuGG6qJEAKbjmRizoZjOHO5GADQwlODcXe0xuO3tYKnliGHiEhJDDe1YLih2lSYzPh+XxoWbD2F1NwSAICfhyue6tcaj8e34ukqIiKFMNzUguGG6qLcZMYPKZew4NeTOJdjCTmeGjUevjUCT/SJRoS/h8IVEhE1Lww3tWC4oZtRYTLj57/SsWjbKZzItFxuQCUBiZ1C8FS/aPSI9IMkSQpXSUTk/BhuasFwQ/UhhMDvJ7Pxn+1n8duJy/LyTmE6PBoXifu6hsGbp6yIiBoNw00tGG6ooY5nFGLp9rNYnZIGY4UZAOChUeO+rmF4NC4SXVr6KlsgEZETYripBcMN2UpesRH/b99FfLMrFacrZ1gBQMdQHR7oEY77uoYhSFfzDV2JiOjmMNzUguGGbE0IgV1nc/H1rlSsP5gBo8nSm6OSgD5tAjC8WzgSO4fAi9PJiYjqjeGmFgw31Jjyio34+WA61uxPw97KiwIClqsf390hGIM7h2BA+yAGHSKim8RwUwuGG2oq53OK8UPKJazZn4Yz2VdOW2lcVLijbQASO4UgoUMw/Dw1ClZJROQYGG5qwXBDTU0Igb8uFmD9oQxsOJQuXzcHANQqCXHR/rizfRAGtA9EmyAvTi0nIqoBw00tGG5ISUIInMgswoZDGdhwOANH0/VW68N93dG/fSD6twtEnzYBPH1FRFSJ4aYWDDdkT87nFGPL0SxsO3EZO8/kyFPLAcBVLaFHpB/iY1ogvnULdIv0hdZFrWC1RETKYbipBcMN2atSownJZ7Kx7fhlbDt+Wb63VRU3VxV6tvLDbdEtEB/TAl1a+kLjolKoWiKipsVwUwuGG3IEQgiczS7GH6dzkHwmB3+eyUF2kdGqjZurCl1a+qJHpB96RPqiRys/BHhpFaqYiKhxMdzUguGGHJEQAqeyipB8JgfJp3Ow80wO8krKq7WL9PeQg073CD+0C/HiqSwicgoMN7VguCFnYDYLnMkuwr7z+diXmod9qXnyjT2v5qqW0C7YG53DfNA5XIdO4T7oGKqDmysDDxE5FoabWjDckLMqKC1HyoV87DtvCTsH0wqQX0PvjloloU2gFzqF6xAb4o12wd5oH+KNEJ0bp6ETkd1iuKkFww01F0IIpOWX4lCaHocvFeBQWgEOpumRXWSosb23mwvaB3ujXYg32gV5oV2IN9oHe6MFx/EQkR1guKkFww01Z0IIZBUacCitAIcv6XE8sxAnMgpxJrsYJnPNfwr8PFwRHeCJ6AAvtA70ROsAT0QHeiKqhSdPbxFRk2G4qQXDDVF1hgoTzmYX43hGIU5kFuJ4RhFOZBbiQl4JavsLEe7rXhl8PBEV4IkIP3dE+HugpZ87vN1cm24HiMjpMdzUguGGqO5KjBU4m11seVy2fD2TXYwzl4ugL6uodVtfD1dE+Hkgwt8dEX4eaOnvIYefcF939voQ0U25mc9vXtudiK7LQ+OCTmE+6BTmY7VcCIG8knKczS7CmcrQcz6nBBfySnAhtwR5JeXILylHfkkBDqYV1PjaLTw1CPFxQ6iPG0J93OXvQ6qe69zgrmEAIqKbx3BDRDdNkiT4e2rg7+mPnq38q60vMlTgQq4l6FzIK8WF3BJczCvBhdxSXMgrQYnRhJxiI3KKjTh8SV/DO1j4ebgixMcdoT5uCNZpEejthkBvLQK9tAj01iDQy/KcIYiIrsZwQ0Q256V1QYdQHTqEVu86FkIgv6Qc6QVlyNCX4lJ+GTIKyuTn6QVlSM8vQ2m5CXkl5cgrKa92g9Ga3u9K6Lnq4aVFCy9NZRDTwM9TA2+tC6e8Ezk5hhsialKSJMGvMmh0DKv5vLkQAvrSCqRfFXayCstwudBgeRQZ5O8NFWYUGSpQZLCMD7oRV7UEP4/KsFP11dMV/p5a+Hu4ws/zyroWXhr4umvg5qpiICJyIAw3RGR3JEmCj4crfDxcERty/YGDQggUGirkoJN9Vei5XGhAVqEBucVG+VFabkK5yTIdPquw5uv91ESjVkHn7gofdxf4uLtWe+gqHzWt89CoGYyImhjDDRE5LEmSoHNzhc7NFTGBXjdsX2o0Ia/EEnSqvuYWG5FXbERuiRF5xeXIKTYgr7i88rkRFWYBo8mM7CLDdS+AWBtXtaVGLzcXeGktD283V3hXPXerWnZlvZebC7y1rlbrtC7sPSKqK4YbImo23DVquGvcEebrXqf2QggUG00oKC1HQUm55WtpOfSlV76/9qEvu7K+3CRQbhLy4OmGcFFJVgHJQ6OGh8by1VPrAneNGp4aNdw1LvDUqK3We8jtLcs8K5e5u6qhVjEwkfNhuCEiug5JkuQwEV7HQFRFCIHScpMceooNFSgss4wNKiqzfF9Y+X2RoRxF16yXvxorIARQYRaV0+ur3y+sIdxcVVdCUGU4cndVwc1VDXdXNdzkx9XLVHB3VUNbue7qZVe3dbtqPUMUNSWGGyKiRiBJUmVocEGoz80Fo6uZzQLFxqtCUeXXEqMJpeUVKDaYUGo0odhYIX8tMZpQYjChpNyEEkPl86rlld9X3W2jrNyMsnIjcm88FrtBXNWSVVCqCkJaFxU0LipoXdTQqFXQuqqu+qq+5rkK2sptqh7ytrU+t2zL03rNB8MNEZEdU6mkyjE6roDPjdvXhRAChgozig3Wgafq+7JyE0rLTTBUfrUEIOvvqx7Vl5krl5lgqDDL72k5RWfpnVKK5upQpFbB1UUFV7XloVFL8veuLtc8V6ugcbnmedV6l5q3d1VdvV6yvN91X0sFVxcJLipLW4awhmO4ISJqZiTpSi9Ki0Z8H7PZEqKuhKAr4acqOBkrzDCazDCUm2EwmWEoN8nPr3w1XfPcDENFDcuu2tZgMsN4VbgCYHmvCjMKG3GfbUElAS5qFVxUElwqQ5K68quLWrJ8r7J876KSrrRVXwlI6srlrioJ6quWuVa2Vauv8xpV26mvbOeiqnr/a9tdvc7y1UVleR83VzUCvbWK/QwZboiIqFGoVFLlIG41/BR4fyFEZRCyhJorX00or7CsK7/qYawQ1s9NAuUV1zw3meVl8vM6bl9uuuo9K648v5ZZVAYxBX5mttI90herJ/ZR7P0ZboiIyClJkgStixpaF/u9PYcQAhVmIYcfk1mgwmRGuVnAZBIoN5tRYRKoqPbVso3JbJmRV2G+6nt5e3NlOwGT2Sy3qzALy2uYrnxfXrl9hemq1zVf3eaq7a6qocJkhklU1l1Zc4VZQOuiUvTnynBDRESkEEmynO5xVSsbBpwNf5pERETkVBhuiIiIyKkw3BAREZFTYbghIiIip8JwQ0RERE6F4YaIiIicCsMNERERORWGGyIiInIqDDdERETkVBhuiIiIyKkw3BAREZFTYbghIiIip8JwQ0RERE6F4YaIiIiciovSBTQ1IQQAQK/XK1wJERER1VXV53bV53html24KSwsBABEREQoXAkRERHdrMLCQvj4+NTaRhJ1iUBOxGw249KlS/D29oYkSTZ9bb1ej4iICFy4cAE6nc6mr20PnH3/AOffR+6f43P2feT+Ob7G2kchBAoLCxEWFgaVqvZRNc2u50alUqFly5aN+h46nc5pf2kB598/wPn3kfvn+Jx9H7l/jq8x9vFGPTZVOKCYiIiInArDDRERETkVhhsb0mq1mD59OrRardKlNApn3z/A+feR++f4nH0fuX+Ozx72sdkNKCYiIiLnxp4bIiIicioMN0RERORUGG6IiIjIqTDcEBERkVNhuLGRhQsXIioqCm5uboiLi8OuXbuULqlGs2fPxq233gpvb28EBQVh+PDhOH78uFWbAQMGQJIkq8czzzxj1SY1NRVDhgyBh4cHgoKC8I9//AMVFRVWbbZt24YePXpAq9WiTZs2WL58eWPvHmbMmFGt9tjYWHl9WVkZJk2ahBYtWsDLywsPPvggMjMzHWLfqkRFRVXbR0mSMGnSJACOd/x+++03DB06FGFhYZAkCWvWrLFaL4TAtGnTEBoaCnd3dyQkJODkyZNWbXJzczFq1CjodDr4+vriySefRFFRkVWbv/76C/369YObmxsiIiLw3nvvVavl22+/RWxsLNzc3HDLLbdg3bp1jbp/5eXlePXVV3HLLbfA09MTYWFhGD16NC5dumT1GjUd8zlz5tjF/t1oHwFgzJgx1eofNGiQVRtHPYYAavz3KEkS5s6dK7ex52NYl8+FpvzbaZPPU0ENtmLFCqHRaMTSpUvF4cOHxbhx44Svr6/IzMxUurRqEhMTxbJly8ShQ4dESkqKuPfee0VkZKQoKiqS2/Tv31+MGzdOpKeny4+CggJ5fUVFhejcubNISEgQ+/fvF+vWrRMBAQFi6tSpcpszZ84IDw8PMWXKFHHkyBHx0UcfCbVaLTZs2NCo+zd9+nTRqVMnq9ovX74sr3/mmWdERESE2LJli9izZ4+47bbbxO233+4Q+1YlKyvLav82bdokAIitW7cKIRzv+K1bt068/vrr4vvvvxcAxOrVq63Wz5kzR/j4+Ig1a9aIAwcOiPvuu09ER0eL0tJSuc2gQYNE165dxc6dO8Xvv/8u2rRpI0aOHCmvLygoEMHBwWLUqFHi0KFD4ptvvhHu7u7i448/ltvs2LFDqNVq8d5774kjR46IN954Q7i6uoqDBw822v7l5+eLhIQEsXLlSnHs2DGRnJwsevfuLXr27Gn1Gq1atRKzZs2yOqZX/5tVcv9utI9CCJGUlCQGDRpkVX9ubq5VG0c9hkIIq/1KT08XS5cuFZIkidOnT8tt7PkY1uVzoan+dtrq85ThxgZ69+4tJk2aJD83mUwiLCxMzJ49W8Gq6iYrK0sAEP/73//kZf379xfPP//8dbdZt26dUKlUIiMjQ162ePFiodPphMFgEEII8corr4hOnTpZbTdixAiRmJho2x24xvTp00XXrl1rXJefny9cXV3Ft99+Ky87evSoACCSk5OFEPa9b9fz/PPPi5iYGGE2m4UQjn38rv3gMJvNIiQkRMydO1delp+fL7Rarfjmm2+EEEIcOXJEABC7d++W26xfv15IkiTS0tKEEEIsWrRI+Pn5yfsnhBCvvvqqaN++vfz84YcfFkOGDLGqJy4uTjz99NONtn812bVrlwAgzp8/Ly9r1aqV+PDDD6+7jb3snxA172NSUpIYNmzYdbdxtmM4bNgwcdddd1ktc6RjeO3nQlP+7bTV5ylPSzWQ0WjE3r17kZCQIC9TqVRISEhAcnKygpXVTUFBAQDA39/favlXX32FgIAAdO7cGVOnTkVJSYm8Ljk5GbfccguCg4PlZYmJidDr9Th8+LDc5uqfSVWbpviZnDx5EmFhYWjdujVGjRqF1NRUAMDevXtRXl5uVVdsbCwiIyPluux9365lNBrx3//+F0888YTVjWAd+fhd7ezZs8jIyLCqxcfHB3FxcVbHzNfXF7169ZLbJCQkQKVS4c8//5Tb3HHHHdBoNHKbxMREHD9+HHl5eXIbe9jngoICSJIEX19fq+Vz5sxBixYt0L17d8ydO9equ98R9m/btm0ICgpC+/btMWHCBOTk5FjV7yzHMDMzE2vXrsWTTz5ZbZ2jHMNrPxea6m+nLT9Pm92NM20tOzsbJpPJ6oACQHBwMI4dO6ZQVXVjNpvxwgsvoE+fPujcubO8/NFHH0WrVq0QFhaGv/76C6+++iqOHz+O77//HgCQkZFR4/5WrautjV6vR2lpKdzd3Rtln+Li4rB8+XK0b98e6enpmDlzJvr164dDhw4hIyMDGo2m2odGcHDwDeu2h32ryZo1a5Cfn48xY8bIyxz5+F2rqp6aarm61qCgIKv1Li4u8Pf3t2oTHR1d7TWq1vn5+V13n6teoymUlZXh1VdfxciRI61uOPjcc8+hR48e8Pf3xx9//IGpU6ciPT0dH3zwgbwP9rx/gwYNwgMPPIDo6GicPn0a//znPzF48GAkJydDrVY71TH8/PPP4e3tjQceeMBquaMcw5o+F5rqb2deXp7NPk8ZbpqxSZMm4dChQ9i+fbvV8vHjx8vf33LLLQgNDcXdd9+N06dPIyYmpqnLvCmDBw+Wv+/SpQvi4uLQqlUrrFq1qklDR1P57LPPMHjwYISFhcnLHPn4NWfl5eV4+OGHIYTA4sWLrdZNmTJF/r5Lly7QaDR4+umnMXv2bIe4jP8jjzwif3/LLbegS5cuiImJwbZt23D33XcrWJntLV26FKNGjYKbm5vVckc5htf7XHA0PC3VQAEBAVCr1dVGjWdmZiIkJEShqm5s8uTJ+Pnnn7F161a0bNmy1rZxcXEAgFOnTgEAQkJCatzfqnW1tdHpdE0aMnx9fdGuXTucOnUKISEhMBqNyM/Pr1bXjequWldbm6bet/Pnz2Pz5s146qmnam3nyMevqp7a/n2FhIQgKyvLan1FRQVyc3Ntclyb4t9xVbA5f/48Nm3aZNVrU5O4uDhUVFTg3LlzAOx//67VunVrBAQEWP1OOvoxBIDff/8dx48fv+G/ScA+j+H1Phea6m+nLT9PGW4aSKPRoGfPntiyZYu8zGw2Y8uWLYiPj1ewspoJITB58mSsXr0av/76a7Vu0JqkpKQAAEJDQwEA8fHxOHjwoNUfo6o/yB07dpTbXP0zqWrT1D+ToqIinD59GqGhoejZsydcXV2t6jp+/DhSU1Pluhxp35YtW4agoCAMGTKk1naOfPyio6MREhJiVYter8eff/5pdczy8/Oxd+9euc2vv/4Ks9ksB7v4+Hj89ttvKC8vl9ts2rQJ7du3h5+fn9xGiX2uCjYnT57E5s2b0aJFixtuk5KSApVKJZ/Ksef9q8nFixeRk5Nj9TvpyMewymeffYaePXuia9euN2xrT8fwRp8LTfW306afpzc1/JhqtGLFCqHVasXy5cvFkSNHxPjx44Wvr6/VqHF7MWHCBOHj4yO2bdtmNSWxpKRECCHEqVOnxKxZs8SePXvE2bNnxQ8//CBat24t7rjjDvk1qqb8DRw4UKSkpIgNGzaIwMDAGqf8/eMf/xBHjx4VCxcubJLp0i+99JLYtm2bOHv2rNixY4dISEgQAQEBIisrSwhhmc4YGRkpfv31V7Fnzx4RHx8v4uPjHWLfrmYymURkZKR49dVXrZY74vErLCwU+/fvF/v37xcAxAcffCD2798vzxaaM2eO8PX1FT/88IP466+/xLBhw2qcCt69e3fx559/iu3bt4u2bdtaTSPOz88XwcHB4vHHHxeHDh0SK1asEB4eHtWm2bq4uIh58+aJo0ePiunTp9tkmm1t+2c0GsV9990nWrZsKVJSUqz+TVbNMPnjjz/Ehx9+KFJSUsTp06fFf//7XxEYGChGjx5tF/t3o30sLCwUL7/8skhOThZnz54VmzdvFj169BBt27YVZWVl8ms46jGsUlBQIDw8PMTixYurbW/vx/BGnwtCNN3fTlt9njLc2MhHH30kIiMjhUajEb179xY7d+5UuqQaAajxsWzZMiGEEKmpqeKOO+4Q/v7+QqvVijZt2oh//OMfVtdJEUKIc+fOicGDBwt3d3cREBAgXnrpJVFeXm7VZuvWraJbt25Co9GI1q1by+/RmEaMGCFCQ0OFRqMR4eHhYsSIEeLUqVPy+tLSUjFx4kTh5+cnPDw8xP333y/S09MdYt+u9ssvvwgA4vjx41bLHfH4bd26tcbfyaSkJCGEZTr4m2++KYKDg4VWqxV33313tf3OyckRI0eOFF5eXkKn04mxY8eKwsJCqzYHDhwQffv2FVqtVoSHh4s5c+ZUq2XVqlWiXbt2QqPRiE6dOom1a9c26v6dPXv2uv8mq65btHfvXhEXFyd8fHyEm5ub6NChg3jnnXesgoGS+3ejfSwpKREDBw4UgYGBwtXVVbRq1UqMGzeu2oeVox7DKh9//LFwd3cX+fn51ba392N4o88FIZr2b6ctPk+lyh0jIiIicgocc0NEREROheGGiIiInArDDRERETkVhhsiIiJyKgw3RERE5FQYboiIiMipMNwQERGRU2G4IaJmT5IkrFmzRukyiMhGGG6ISFFjxoyBJEnVHoMGDVK6NCJyUC5KF0BENGjQICxbtsxqmVarVagaInJ07LkhIsVptVqEhIRYParuhCxJEhYvXozBgwfD3d0drVu3xnfffWe1/cGDB3HXXXfB3d0dLVq0wPjx41FUVGTVZunSpejUqRO0Wi1CQ0MxefJkq/XZ2dm4//774eHhgbZt2+LHH39s3J0mokbDcENEdu/NN9/Egw8+iAMHDmDUqFF45JFHcPToUQBAcXExEhMT4efnh927d+Pbb7/F5s2brcLL4sWLMWnSJIwfPx4HDx7Ejz/+iDZt2li9x8yZM/Hwww/jr7/+wr333otRo0YhNze3SfeTiGzkpm+1SURkQ0lJSUKtVgtPT0+rx9tvvy2EsNyx+JlnnrHaJi4uTkyYMEEIIcQnn3wi/Pz8RFFRkbx+7dq1QqVSyXeeDgsLE6+//vp1awAg3njjDfl5UVGRACDWr19vs/0koqbDMTdEpLg777wTixcvtlrm7+8vfx8fH2+1Lj4+HikpKQCAo0ePomvXrvD09JTX9+nTB2azGcePH4ckSbh06RLuvvvuWmvo0qWL/L2npyd0Oh2ysrLqu0tEpCCGGyJSnKenZ7XTRLbi7u5ep3aurq5WzyVJgtlsboySiKiRccwNEdm9nTt3VnveoUMHAECHDh1w4MABFBcXy+t37NgBlUqF9u3bw9vbG1FRUdiyZUuT1kxEymHPDREpzmAwICMjw2qZi4sLAgICAADffvstevXqhb59++Krr77Crl278NlnnwEARo0ahenTpyMpKQkzZszA5cuX8eyzz+Lxxx9HcHAwAGDGjBl45plnEBQUhMGDB6OwsBA7duzAs88+27Q7SkRNguGGiBS3YcMGhIaGWi1r3749jh07BsAyk2nFihWYOHEiQkND8c0336Bjx44AAA8PD/zyyy94/vnnceutt8LDwwMPPvggPvjgA/m1kpKSUFZWhg8//BAvv/wyAgIC8NBDDzXdDhJRk5KEEELpIoiIrkeSJKxevRrDhw9XuhQichAcc0NEREROheGGiIiInArH3BCRXeOZcyK6Wey5ISIiIqfCcENEREROheGGiIiInArDDRERETkVhhsiIiJyKgw3RERE5FQYboiIiMipMNwQERGRU2G4ISIiIqfy/wGGmvvpVtypIAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "\n",
    "# 仍然用4特征，3分类的鸢尾花数据集作为我们今天的数据集\n",
    "# 加载鸢尾花数据集\n",
    "iris = load_iris()\n",
    "X = iris.data  # 特征数据\n",
    "y = iris.target  # 标签数据\n",
    "# 划分训练集和测试集\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# # 打印下尺寸\n",
    "# print(X_train.shape)\n",
    "# print(y_train.shape)\n",
    "# print(X_test.shape)\n",
    "# print(y_test.shape)\n",
    "\n",
    "# 归一化数据，神经网络对于输入数据的尺寸敏感，归一化是最常见的处理方式\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "scaler = MinMaxScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test) #确保训练集和测试集是相同的缩放\n",
    "\n",
    "\n",
    "# 将数据转换为 PyTorch 张量，因为 PyTorch 使用张量进行训练\n",
    "# y_train和y_test是整数，所以需要转化为long类型，如果是float32，会输出1.0 0.0\n",
    "X_train = torch.FloatTensor(X_train)\n",
    "y_train = torch.LongTensor(y_train)\n",
    "X_test = torch.FloatTensor(X_test)\n",
    "y_test = torch.LongTensor(y_test)\n",
    "\n",
    "class MLP(nn.Module): # 定义一个多层感知机（MLP）模型，继承父类nn.Module\n",
    "    def __init__(self): # 初始化函数\n",
    "        super(MLP, self).__init__() # 调用父类的初始化函数\n",
    " # 前三行是八股文，后面的是自定义的\n",
    "\n",
    "        self.fc1 = nn.Linear(4, 10)  # 输入层到隐藏层\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(10, 3)  # 隐藏层到输出层\n",
    "# 输出层不需要激活函数，因为后面会用到交叉熵函数cross_entropy，交叉熵函数内部有softmax函数，会把输出转化为概率\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.fc1(x)\n",
    "        out = self.relu(out)\n",
    "        out = self.fc2(out)\n",
    "        return out\n",
    "\n",
    "# 实例化模型\n",
    "model = MLP()\n",
    "\n",
    "# 分类问题使用交叉熵损失函数\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# 使用随机梯度下降优化器\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)\n",
    "\n",
    "# # 使用自适应学习率的化器\n",
    "# optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "\n",
    "# 训练模型\n",
    "num_epochs = 20000 # 训练的轮数\n",
    "\n",
    "# 用于存储每个 epoch 的损失值\n",
    "losses = []\n",
    "\n",
    "import time\n",
    "start_time = time.time() # 记录开始时间\n",
    "\n",
    "for epoch in range(num_epochs): # range是从0开始，所以epoch是从0开始\n",
    "    # 前向传播\n",
    "    outputs = model.forward(X_train)   # 显式调用forward函数\n",
    "    # outputs = model(X_train)  # 常见写法隐式调用forward函数，其实是用了model类的__call__方法\n",
    "    loss = criterion(outputs, y_train) # output是模型预测值，y_train是真实标签\n",
    "\n",
    "    # 反向传播和优化\n",
    "    optimizer.zero_grad() #梯度清零，因为PyTorch会累积梯度，所以每次迭代需要清零，梯度累计是那种小的bitchsize模拟大的bitchsize\n",
    "    loss.backward() # 反向传播计算梯度\n",
    "    optimizer.step() # 更新参数\n",
    "\n",
    "    # 记录损失值\n",
    "    losses.append(loss.item())\n",
    "\n",
    "    # 打印训练信息\n",
    "    if (epoch + 1) % 100 == 0: # range是从0开始，所以epoch+1是从当前epoch开始，每100个epoch打印一次\n",
    "        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')\n",
    "\n",
    "time_all = time.time() - start_time # 计算训练时间\n",
    "print(f'Training time: {time_all:.2f} seconds')\n",
    "import matplotlib.pyplot as plt\n",
    "# 可视化损失曲线\n",
    "plt.plot(range(num_epochs), losses)\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.title('Training Loss over Epochs')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5324c1a7",
   "metadata": {},
   "source": [
    "### CPU性能的查看"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "358553dd",
   "metadata": {},
   "source": [
    "上述是在cpu的情况下训练，（即使安装了cuda，但是没有使用cuda），我们借这个机会简单介绍下cpu的性能差异。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "0a70c0a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU 型号: 12th Gen Intel(R) Core(TM) i9-12900KF\n",
      "核心数: 16\n",
      "线程数: 24\n"
     ]
    }
   ],
   "source": [
    "# pip install wmi -i https://pypi.tuna.tsinghua.edu.cn/simple\n",
    "# 这是Windows专用的库，Linux和MacOS不支持，其他系统自行询问大模型\n",
    "# 我想查看一下CPU的型号和核心数\n",
    "import wmi\n",
    "\n",
    "c = wmi.WMI()\n",
    "processors = c.Win32_Processor()\n",
    "\n",
    "for processor in processors:\n",
    "    print(f\"CPU 型号: {processor.Name}\")\n",
    "    print(f\"核心数: {processor.NumberOfCores}\")\n",
    "    print(f\"线程数: {processor.NumberOfLogicalProcessors}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f057298",
   "metadata": {},
   "source": [
    "解读下我的cpu配置\n",
    "- Intel 第 12 代酷睿（Alder Lake 架构，2021 年发布）\n",
    "- K：支持超频（解锁倍频）\n",
    "- F：无内置核显（需搭配独立显卡使用）\n",
    "- 核心架构：\n",
    "  - 性能核（P-Core）：8 核（支持超线程，共 16 线程），擅长单线程高性能任务（如游戏、视频剪辑）\n",
    "  - 能效核（E-Core）：8 核（不支持超线程，共 8 线程），优化多线程能效比（如后台任务、虚拟机）。\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95ccc888",
   "metadata": {},
   "source": [
    "判断 CPU 的好坏需要综合考虑硬件参数、性能表现、适用场景。\n",
    "\n",
    "1. 看架构代际，新一代架构通常优化指令集、缓存设计和能效比。如Intel 第 13 代 i5-13600K 比第 12 代 i5-12600K 多核性能提升约 15%\n",
    "2. 看制程工艺，制程越小，晶体管密度越高，能效比越好，如AMD Ryzen 7000 系列（5nm）比 Ryzen 5000 系列（7nm）能效比提升约 30%。\n",
    "3. 看核心数：性能核负责高负载任务（如游戏、视频剪辑），单核性能强。能效核负责多任务后台处理（如下载、杀毒），功耗低。如游戏 / 办公：4-8 核足够，内容创作 / 编程：12 核以上更优。\n",
    "4. 看线程数目\n",
    "5. 看频率，高频适合单线程任务（如游戏、Office），低频多核适合多线程任务（如 3D 渲染）\n",
    "6. 支持的指令集和扩展能力。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78ede2ca",
   "metadata": {},
   "source": [
    "可以看到在我的电脑的配置下，cpu运行时长，是2.90s\n",
    "\n",
    "昨天的训练是cpu的训练，今天介绍下gpu的训练。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31eb833c",
   "metadata": {},
   "source": [
    "## GPU训练\n",
    "要让模型在 GPU 上训练，主要是将模型和数据迁移到 GPU 设备上。\n",
    "\n",
    "在 PyTorch 里，.to(device) 方法的作用是把张量或者模型转移到指定的计算设备（像 CPU 或者 GPU）上。\n",
    "\n",
    "- 对于张量（Tensor）：调用 .to(device) 之后，会返回一个在新设备上的新张量。\n",
    "- 对于模型（nn.Module）：调用 .to(device) 会直接对模型进行修改，让其所有参数和缓冲区都移到新设备上。\n",
    "\n",
    "在进行计算时，所有输入张量和模型必须处于同一个设备。要是它们不在同一设备上，就会引发运行时错误。并非所有 PyTorch 对象都有 .to(device) 方法，只有继承自 torch.nn.Module 的模型以及 torch.Tensor 对象才有此方法。\n",
    "\n",
    "```\n",
    "RuntimeError: Tensor for argument #1 'input' is on CPU, but expected it to be on GPU\n",
    "```\n",
    "\n",
    "这个常见错误就是输入张量和模型处于不同的设备。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59418a14",
   "metadata": {},
   "source": [
    "如何衡量GPU的性能好坏呢？\n",
    "\n",
    "以RTX 3090 Ti,  RTX 3080, RTX 3070 Ti, RTX 3070, RTX 4070等为例\n",
    "- 通过“代”\n",
    "前两位数字代表“代”: 40xx (第40代), 30xx (第30代), 20xx (第20代)。“代”通常指的是其底层的架构 (Architecture)。每一代新架构的发布，通常会带来工艺制程的进步和其他改进。也就是新一代架构的目标是在能效比和绝对性能上超越前一代同型号的产品。\n",
    "\n",
    "- 通过级别\n",
    "后面的数字代表“级别”，\n",
    "  - xx90: 通常是该代的消费级旗舰或次旗舰，性能最强，显存最大 (如 RTX 4090, RTX 3090)。\n",
    "  - xx80: 高端型号，性能强劲，显存较多 (如 RTX 4080, RTX 3080)。\n",
    "  - xx70: 中高端，甜点级，性能和价格平衡较好 (如 RTX 4070, RTX 3070)。\n",
    "  - xx60: 主流中端，性价比较高，适合入门或预算有限 (如 RTX 4060, RTX 3060)。\n",
    "  - xx50: 入门级，深度学习能力有限。\n",
    "\n",
    "- 通过后缀\n",
    "Ti 通常是同型号的增强版，性能介于原型号和更高一级型号之间 (如 RTX 4070 Ti 强于 RTX 4070，小于4080)。\n",
    "\n",
    "- 通过显存容量 VRAM （最重要！！）\n",
    "他是GPU 自身的独立高速内存，用于存储模型参数、激活值、输入数据批次等。单位通常是 GB（例如 8GB, 12GB, 24GB, 48GB）。如果显存不足，可能无法加载模型，或者被迫使用很小的批量大小，从而影响训练速度和效果\n",
    "\n",
    "1. 训练阶段：小批量梯度是对真实梯度的一个有噪声的估计。批量越小，梯度的方差越大（噪声越大）。显存小只能够使用小批量梯度。\n",
    "2. 推理阶段：有些模型本身就非常庞大（例如大型语言模型、高分辨率图像的复杂 CNN 网络）。即使你将批量大小减到 1，模型参数本身占用的显存可能就已经超出了你的 GPU 显存上限。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "a0985a67",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CUDA可用！\n",
      "可用的CUDA设备数量: 1\n",
      "当前使用的CUDA设备索引: 0\n",
      "当前CUDA设备的名称: NVIDIA GeForce RTX 3080 Ti\n",
      "CUDA版本: 11.1\n",
      "cuDNN版本: 8005\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "# 检查CUDA是否可用\n",
    "if torch.cuda.is_available():\n",
    "    print(\"CUDA可用！\")\n",
    "    # 获取可用的CUDA设备数量\n",
    "    device_count = torch.cuda.device_count()\n",
    "    print(f\"可用的CUDA设备数量: {device_count}\")\n",
    "    # 获取当前使用的CUDA设备索引\n",
    "    current_device = torch.cuda.current_device()\n",
    "    print(f\"当前使用的CUDA设备索引: {current_device}\")\n",
    "    # 获取当前CUDA设备的名称\n",
    "    device_name = torch.cuda.get_device_name(current_device)\n",
    "    print(f\"当前CUDA设备的名称: {device_name}\")\n",
    "    # 获取CUDA版本\n",
    "    cuda_version = torch.version.cuda\n",
    "    print(f\"CUDA版本: {cuda_version}\")\n",
    "    # 查看cuDNN版本（如果可用）\n",
    "    print(\"cuDNN版本:\", torch.backends.cudnn.version())\n",
    "\n",
    "else:\n",
    "    print(\"CUDA不可用。\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "94c43772",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "使用设备: cuda:0\n"
     ]
    }
   ],
   "source": [
    "# 设置GPU设备\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(f\"使用设备: {device}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "bd99fce2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 加载鸢尾花数据集\n",
    "iris = load_iris()\n",
    "X = iris.data  # 特征数据\n",
    "y = iris.target  # 标签数据\n",
    "\n",
    "# 划分训练集和测试集\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# 归一化数据\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "scaler = MinMaxScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)\n",
    "\n",
    "# 将数据转换为PyTorch张量并移至GPU\n",
    "# 分类问题交叉熵损失要求标签为long类型\n",
    "# 张量具有to(device)方法，可以将张量移动到指定的设备上\n",
    "X_train = torch.FloatTensor(X_train).to(device)\n",
    "y_train = torch.LongTensor(y_train).to(device)\n",
    "X_test = torch.FloatTensor(X_test).to(device)\n",
    "y_test = torch.LongTensor(y_test).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "b9790630",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLP(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MLP, self).__init__()\n",
    "        self.fc1 = nn.Linear(4, 10)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(10, 3)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.fc1(x)\n",
    "        out = self.relu(out)\n",
    "        out = self.fc2(out)\n",
    "        return out\n",
    "\n",
    "# 实例化模型并移至GPU\n",
    "# MLP继承nn.Module类，所以也具有to(device)方法\n",
    "model = MLP().to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "0f895cc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义损失函数和优化器\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)\n",
    "\n",
    "# 训练模型\n",
    "num_epochs = 20000\n",
    "losses = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "06555411",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [100/20000], Loss: 1.0763\n",
      "Epoch [200/20000], Loss: 1.0542\n",
      "Epoch [300/20000], Loss: 1.0276\n",
      "Epoch [400/20000], Loss: 0.9927\n",
      "Epoch [500/20000], Loss: 0.9465\n",
      "Epoch [600/20000], Loss: 0.8870\n",
      "Epoch [700/20000], Loss: 0.8205\n",
      "Epoch [800/20000], Loss: 0.7541\n",
      "Epoch [900/20000], Loss: 0.6927\n",
      "Epoch [1000/20000], Loss: 0.6391\n",
      "Epoch [1100/20000], Loss: 0.5941\n",
      "Epoch [1200/20000], Loss: 0.5566\n",
      "Epoch [1300/20000], Loss: 0.5253\n",
      "Epoch [1400/20000], Loss: 0.4984\n",
      "Epoch [1500/20000], Loss: 0.4751\n",
      "Epoch [1600/20000], Loss: 0.4545\n",
      "Epoch [1700/20000], Loss: 0.4360\n",
      "Epoch [1800/20000], Loss: 0.4189\n",
      "Epoch [1900/20000], Loss: 0.4032\n",
      "Epoch [2000/20000], Loss: 0.3884\n",
      "Epoch [2100/20000], Loss: 0.3746\n",
      "Epoch [2200/20000], Loss: 0.3614\n",
      "Epoch [2300/20000], Loss: 0.3487\n",
      "Epoch [2400/20000], Loss: 0.3367\n",
      "Epoch [2500/20000], Loss: 0.3251\n",
      "Epoch [2600/20000], Loss: 0.3140\n",
      "Epoch [2700/20000], Loss: 0.3034\n",
      "Epoch [2800/20000], Loss: 0.2933\n",
      "Epoch [2900/20000], Loss: 0.2835\n",
      "Epoch [3000/20000], Loss: 0.2742\n",
      "Epoch [3100/20000], Loss: 0.2654\n",
      "Epoch [3200/20000], Loss: 0.2570\n",
      "Epoch [3300/20000], Loss: 0.2490\n",
      "Epoch [3400/20000], Loss: 0.2413\n",
      "Epoch [3500/20000], Loss: 0.2340\n",
      "Epoch [3600/20000], Loss: 0.2270\n",
      "Epoch [3700/20000], Loss: 0.2204\n",
      "Epoch [3800/20000], Loss: 0.2142\n",
      "Epoch [3900/20000], Loss: 0.2082\n",
      "Epoch [4000/20000], Loss: 0.2026\n",
      "Epoch [4100/20000], Loss: 0.1972\n",
      "Epoch [4200/20000], Loss: 0.1921\n",
      "Epoch [4300/20000], Loss: 0.1872\n",
      "Epoch [4400/20000], Loss: 0.1826\n",
      "Epoch [4500/20000], Loss: 0.1782\n",
      "Epoch [4600/20000], Loss: 0.1740\n",
      "Epoch [4700/20000], Loss: 0.1701\n",
      "Epoch [4800/20000], Loss: 0.1663\n",
      "Epoch [4900/20000], Loss: 0.1627\n",
      "Epoch [5000/20000], Loss: 0.1593\n",
      "Epoch [5100/20000], Loss: 0.1560\n",
      "Epoch [5200/20000], Loss: 0.1529\n",
      "Epoch [5300/20000], Loss: 0.1498\n",
      "Epoch [5400/20000], Loss: 0.1470\n",
      "Epoch [5500/20000], Loss: 0.1442\n",
      "Epoch [5600/20000], Loss: 0.1416\n",
      "Epoch [5700/20000], Loss: 0.1391\n",
      "Epoch [5800/20000], Loss: 0.1368\n",
      "Epoch [5900/20000], Loss: 0.1344\n",
      "Epoch [6000/20000], Loss: 0.1322\n",
      "Epoch [6100/20000], Loss: 0.1301\n",
      "Epoch [6200/20000], Loss: 0.1281\n",
      "Epoch [6300/20000], Loss: 0.1261\n",
      "Epoch [6400/20000], Loss: 0.1243\n",
      "Epoch [6500/20000], Loss: 0.1224\n",
      "Epoch [6600/20000], Loss: 0.1207\n",
      "Epoch [6700/20000], Loss: 0.1190\n",
      "Epoch [6800/20000], Loss: 0.1175\n",
      "Epoch [6900/20000], Loss: 0.1159\n",
      "Epoch [7000/20000], Loss: 0.1144\n",
      "Epoch [7100/20000], Loss: 0.1130\n",
      "Epoch [7200/20000], Loss: 0.1115\n",
      "Epoch [7300/20000], Loss: 0.1102\n",
      "Epoch [7400/20000], Loss: 0.1089\n",
      "Epoch [7500/20000], Loss: 0.1077\n",
      "Epoch [7600/20000], Loss: 0.1065\n",
      "Epoch [7700/20000], Loss: 0.1053\n",
      "Epoch [7800/20000], Loss: 0.1041\n",
      "Epoch [7900/20000], Loss: 0.1031\n",
      "Epoch [8000/20000], Loss: 0.1020\n",
      "Epoch [8100/20000], Loss: 0.1010\n",
      "Epoch [8200/20000], Loss: 0.1000\n",
      "Epoch [8300/20000], Loss: 0.0990\n",
      "Epoch [8400/20000], Loss: 0.0981\n",
      "Epoch [8500/20000], Loss: 0.0972\n",
      "Epoch [8600/20000], Loss: 0.0963\n",
      "Epoch [8700/20000], Loss: 0.0954\n",
      "Epoch [8800/20000], Loss: 0.0945\n",
      "Epoch [8900/20000], Loss: 0.0938\n",
      "Epoch [9000/20000], Loss: 0.0930\n",
      "Epoch [9100/20000], Loss: 0.0922\n",
      "Epoch [9200/20000], Loss: 0.0915\n",
      "Epoch [9300/20000], Loss: 0.0908\n",
      "Epoch [9400/20000], Loss: 0.0901\n",
      "Epoch [9500/20000], Loss: 0.0894\n",
      "Epoch [9600/20000], Loss: 0.0887\n",
      "Epoch [9700/20000], Loss: 0.0881\n",
      "Epoch [9800/20000], Loss: 0.0874\n",
      "Epoch [9900/20000], Loss: 0.0868\n",
      "Epoch [10000/20000], Loss: 0.0863\n",
      "Epoch [10100/20000], Loss: 0.0857\n",
      "Epoch [10200/20000], Loss: 0.0851\n",
      "Epoch [10300/20000], Loss: 0.0846\n",
      "Epoch [10400/20000], Loss: 0.0840\n",
      "Epoch [10500/20000], Loss: 0.0835\n",
      "Epoch [10600/20000], Loss: 0.0830\n",
      "Epoch [10700/20000], Loss: 0.0825\n",
      "Epoch [10800/20000], Loss: 0.0820\n",
      "Epoch [10900/20000], Loss: 0.0815\n",
      "Epoch [11000/20000], Loss: 0.0811\n",
      "Epoch [11100/20000], Loss: 0.0806\n",
      "Epoch [11200/20000], Loss: 0.0802\n",
      "Epoch [11300/20000], Loss: 0.0798\n",
      "Epoch [11400/20000], Loss: 0.0793\n",
      "Epoch [11500/20000], Loss: 0.0789\n",
      "Epoch [11600/20000], Loss: 0.0785\n",
      "Epoch [11700/20000], Loss: 0.0781\n",
      "Epoch [11800/20000], Loss: 0.0777\n",
      "Epoch [11900/20000], Loss: 0.0773\n",
      "Epoch [12000/20000], Loss: 0.0770\n",
      "Epoch [12100/20000], Loss: 0.0766\n",
      "Epoch [12200/20000], Loss: 0.0763\n",
      "Epoch [12300/20000], Loss: 0.0759\n",
      "Epoch [12400/20000], Loss: 0.0756\n",
      "Epoch [12500/20000], Loss: 0.0752\n",
      "Epoch [12600/20000], Loss: 0.0749\n",
      "Epoch [12700/20000], Loss: 0.0746\n",
      "Epoch [12800/20000], Loss: 0.0743\n",
      "Epoch [12900/20000], Loss: 0.0740\n",
      "Epoch [13000/20000], Loss: 0.0737\n",
      "Epoch [13100/20000], Loss: 0.0734\n",
      "Epoch [13200/20000], Loss: 0.0731\n",
      "Epoch [13300/20000], Loss: 0.0728\n",
      "Epoch [13400/20000], Loss: 0.0725\n",
      "Epoch [13500/20000], Loss: 0.0722\n",
      "Epoch [13600/20000], Loss: 0.0720\n",
      "Epoch [13700/20000], Loss: 0.0717\n",
      "Epoch [13800/20000], Loss: 0.0715\n",
      "Epoch [13900/20000], Loss: 0.0712\n",
      "Epoch [14000/20000], Loss: 0.0710\n",
      "Epoch [14100/20000], Loss: 0.0707\n",
      "Epoch [14200/20000], Loss: 0.0705\n",
      "Epoch [14300/20000], Loss: 0.0702\n",
      "Epoch [14400/20000], Loss: 0.0700\n",
      "Epoch [14500/20000], Loss: 0.0698\n",
      "Epoch [14600/20000], Loss: 0.0695\n",
      "Epoch [14700/20000], Loss: 0.0693\n",
      "Epoch [14800/20000], Loss: 0.0691\n",
      "Epoch [14900/20000], Loss: 0.0689\n",
      "Epoch [15000/20000], Loss: 0.0687\n",
      "Epoch [15100/20000], Loss: 0.0685\n",
      "Epoch [15200/20000], Loss: 0.0683\n",
      "Epoch [15300/20000], Loss: 0.0681\n",
      "Epoch [15400/20000], Loss: 0.0679\n",
      "Epoch [15500/20000], Loss: 0.0677\n",
      "Epoch [15600/20000], Loss: 0.0675\n",
      "Epoch [15700/20000], Loss: 0.0673\n",
      "Epoch [15800/20000], Loss: 0.0671\n",
      "Epoch [15900/20000], Loss: 0.0669\n",
      "Epoch [16000/20000], Loss: 0.0667\n",
      "Epoch [16100/20000], Loss: 0.0666\n",
      "Epoch [16200/20000], Loss: 0.0664\n",
      "Epoch [16300/20000], Loss: 0.0662\n",
      "Epoch [16400/20000], Loss: 0.0661\n",
      "Epoch [16500/20000], Loss: 0.0659\n",
      "Epoch [16600/20000], Loss: 0.0658\n",
      "Epoch [16700/20000], Loss: 0.0656\n",
      "Epoch [16800/20000], Loss: 0.0655\n",
      "Epoch [16900/20000], Loss: 0.0653\n",
      "Epoch [17000/20000], Loss: 0.0651\n",
      "Epoch [17100/20000], Loss: 0.0650\n",
      "Epoch [17200/20000], Loss: 0.0649\n",
      "Epoch [17300/20000], Loss: 0.0647\n",
      "Epoch [17400/20000], Loss: 0.0646\n",
      "Epoch [17500/20000], Loss: 0.0644\n",
      "Epoch [17600/20000], Loss: 0.0643\n",
      "Epoch [17700/20000], Loss: 0.0641\n",
      "Epoch [17800/20000], Loss: 0.0640\n",
      "Epoch [17900/20000], Loss: 0.0638\n",
      "Epoch [18000/20000], Loss: 0.0637\n",
      "Epoch [18100/20000], Loss: 0.0636\n",
      "Epoch [18200/20000], Loss: 0.0634\n",
      "Epoch [18300/20000], Loss: 0.0633\n",
      "Epoch [18400/20000], Loss: 0.0632\n",
      "Epoch [18500/20000], Loss: 0.0630\n",
      "Epoch [18600/20000], Loss: 0.0629\n",
      "Epoch [18700/20000], Loss: 0.0628\n",
      "Epoch [18800/20000], Loss: 0.0627\n",
      "Epoch [18900/20000], Loss: 0.0626\n",
      "Epoch [19000/20000], Loss: 0.0625\n",
      "Epoch [19100/20000], Loss: 0.0623\n",
      "Epoch [19200/20000], Loss: 0.0622\n",
      "Epoch [19300/20000], Loss: 0.0621\n",
      "Epoch [19400/20000], Loss: 0.0620\n",
      "Epoch [19500/20000], Loss: 0.0619\n",
      "Epoch [19600/20000], Loss: 0.0618\n",
      "Epoch [19700/20000], Loss: 0.0617\n",
      "Epoch [19800/20000], Loss: 0.0616\n",
      "Epoch [19900/20000], Loss: 0.0615\n",
      "Epoch [20000/20000], Loss: 0.0614\n",
      "Training time: 11.29 seconds\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/xnp5ZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABPwUlEQVR4nO3dd3hUVf4G8HdKpqTMpFcCCYTea4yAoEYDsiigKyIrxYIUXRUrq1L0p2BDdxVBXSm6roKsoi5FQ1sEo9QgNfSEkkJIr5PMnN8fk7kwJqROcmcm7+d55snMuefefG9uyLyce+4dhRBCgIiIiMhNKOUugIiIiMiRGG6IiIjIrTDcEBERkVthuCEiIiK3wnBDREREboXhhoiIiNwKww0RERG5FYYbIiIicisMN0RERORWGG6InMiUKVMQFRXVqHXnz58PhULh2IKIarBy5UooFArs3btX7lKIasRwQ1QPCoWiXo/t27fLXaospkyZAm9vb7nLcBu28HC9x6+//ip3iUROTS13AUSu4PPPP7d7/dlnnyExMbFae9euXZv0fT755BNYLJZGrfvSSy/hhRdeaNL3J+fyyiuvIDo6ulp7TEyMDNUQuQ6GG6J6+Mtf/mL3+tdff0ViYmK19j8qKSmBp6dnvb+Ph4dHo+oDALVaDbWa/6RdRXFxMby8vGrtM3LkSAwYMKCFKiJyHzwtReQgw4cPR48ePbBv3z7cdNNN8PT0xN/+9jcAwHfffYdRo0YhPDwcWq0WHTp0wKuvvgqz2Wy3jT/OuTl37hwUCgXefvttfPzxx+jQoQO0Wi0GDhyIPXv22K1b05wbhUKBxx57DOvWrUOPHj2g1WrRvXt3bNq0qVr927dvx4ABA6DT6dChQwd89NFHDp/H8/XXX6N///7Q6/UIDAzEX/7yF1y8eNGuT0ZGBqZOnYo2bdpAq9UiLCwMd911F86dOyf12bt3LxISEhAYGAi9Xo/o6Gg8+OCD9arhww8/RPfu3aHVahEeHo5Zs2YhLy9PWv7YY4/B29sbJSUl1dadMGECQkND7Y7bxo0bMXToUHh5ecHHxwejRo3CkSNH7NaznbY7ffo07rjjDvj4+GDixIn1qrc21/5+vPvuu2jXrh30ej2GDRuGw4cPV+u/detWqVZfX1/cddddOHbsWLV+Fy9exEMPPST9vkZHR2PGjBkwmUx2/crLyzF79mwEBQXBy8sLY8eOxeXLl+36NOVYETUW/5tH5EBXrlzByJEjcd999+Evf/kLQkJCAFjnUHh7e2P27Nnw9vbG1q1bMXfuXBQUFOCtt96qc7v//ve/UVhYiEcffRQKhQJvvvkmxo0bhzNnztQ52rNz50588803mDlzJnx8fPCPf/wDd999N9LS0hAQEAAAOHDgAEaMGIGwsDAsWLAAZrMZr7zyCoKCgpr+Q6mycuVKTJ06FQMHDsTChQuRmZmJv//979i1axcOHDgAX19fAMDdd9+NI0eO4PHHH0dUVBSysrKQmJiItLQ06fXtt9+OoKAgvPDCC/D19cW5c+fwzTff1FnD/PnzsWDBAsTHx2PGjBlISUnB0qVLsWfPHuzatQseHh4YP348lixZgvXr1+PPf/6ztG5JSQl++OEHTJkyBSqVCoD1dOXkyZORkJCAN954AyUlJVi6dCmGDBmCAwcO2AXVyspKJCQkYMiQIXj77bfrNaKXn5+P7OxsuzaFQiEdN5vPPvsMhYWFmDVrFsrKyvD3v/8dt9xyCw4dOiT9Dm7evBkjR45E+/btMX/+fJSWluL999/H4MGDsX//fqnWS5cuYdCgQcjLy8O0adPQpUsXXLx4EWvXrkVJSQk0Go30fR9//HH4+flh3rx5OHfuHN577z089thjWL16NQA06VgRNYkgogabNWuW+OM/n2HDhgkAYtmyZdX6l5SUVGt79NFHhaenpygrK5PaJk+eLNq1aye9Pnv2rAAgAgICRE5OjtT+3XffCQDihx9+kNrmzZtXrSYAQqPRiFOnTkltBw8eFADE+++/L7WNHj1aeHp6iosXL0ptJ0+eFGq1uto2azJ58mTh5eV13eUmk0kEBweLHj16iNLSUqn9v//9rwAg5s6dK4QQIjc3VwAQb7311nW39e233woAYs+ePXXWda2srCyh0WjE7bffLsxms9T+wQcfCABi+fLlQgghLBaLiIiIEHfffbfd+mvWrBEAxI4dO4QQQhQWFgpfX1/xyCOP2PXLyMgQRqPRrn3y5MkCgHjhhRfqVeuKFSsEgBofWq1W6mf7/dDr9eLChQtS+2+//SYAiKeeekpq69OnjwgODhZXrlyR2g4ePCiUSqWYNGmS1DZp0iShVCpr/PlaLBa7+uLj46U2IYR46qmnhEqlEnl5eUKIxh8roqbiaSkiB9JqtZg6dWq1dr1eLz0vLCxEdnY2hg4dipKSEhw/frzO7Y4fPx5+fn7S66FDhwIAzpw5U+e68fHx6NChg/S6V69eMBgM0rpmsxmbN2/GmDFjEB4eLvWLiYnByJEj69x+fezduxdZWVmYOXMmdDqd1D5q1Ch06dIF69evB2D9OWk0Gmzfvh25ubk1bss2wvPf//4XFRUV9a5h8+bNMJlMePLJJ6FUXv3T98gjj8BgMEg1KBQK/PnPf8aGDRtQVFQk9Vu9ejUiIiIwZMgQAEBiYiLy8vIwYcIEZGdnSw+VSoXY2Fhs27atWg0zZsyod70AsGTJEiQmJto9Nm7cWK3fmDFjEBERIb0eNGgQYmNjsWHDBgBAeno6kpOTMWXKFPj7+0v9evXqhdtuu03qZ7FYsG7dOowePbrGuT5/PEU5bdo0u7ahQ4fCbDYjNTUVQOOPFVFTMdwQOVBERITdsL3NkSNHMHbsWBiNRhgMBgQFBUmTkfPz8+vcbtu2be1e24LO9QJAbeva1retm5WVhdLS0hqvwHHUVTm2N7vOnTtXW9alSxdpuVarxRtvvIGNGzciJCQEN910E958801kZGRI/YcNG4a7774bCxYsQGBgIO666y6sWLEC5eXljapBo9Ggffv20nLAGiZLS0vx/fffAwCKioqwYcMG/PnPf5bezE+ePAkAuOWWWxAUFGT3+Omnn5CVlWX3fdRqNdq0aVP3D+sagwYNQnx8vN3j5ptvrtavY8eO1do6deokzVOq7efftWtXZGdno7i4GJcvX0ZBQQF69OhRr/rq+r1s7LEiaiqGGyIHunaExiYvLw/Dhg3DwYMH8corr+CHH35AYmIi3njjDQCo16XftjkefySEaNZ15fDkk0/ixIkTWLhwIXQ6HV5++WV07doVBw4cAGAdPVi7di2SkpLw2GOP4eLFi3jwwQfRv39/u5GWprjhhhsQFRWFNWvWAAB++OEHlJaWYvz48VIf23H7/PPPq42uJCYm4rvvvrPbplartRsxcgd1/W61xLEiqol7/UsjckLbt2/HlStXsHLlSjzxxBP405/+hPj4eLvTTHIKDg6GTqfDqVOnqi2rqa0x2rVrBwBISUmptiwlJUVabtOhQwc8/fTT+Omnn3D48GGYTCa88847dn1uuOEGvPbaa9i7dy+++OILHDlyBF999VWDazCZTDh79my1Gu69915s2rQJBQUFWL16NaKionDDDTfY1QhYf35/HF2Jj4/H8OHD6/ipOI5tFOlaJ06ckCYJ1/bzP378OAIDA+Hl5YWgoCAYDIYar7RqioYeK6KmYrghama2/91eO1JiMpnw4YcfylWSHZVKhfj4eKxbtw6XLl2S2k+dOlXj/I7GGDBgAIKDg7Fs2TK7UxIbN27EsWPHMGrUKADWK5LKysrs1u3QoQN8fHyk9XJzc6uNOvXp0wcAaj3dER8fD41Gg3/84x9263/66afIz8+XarAZP348ysvLsWrVKmzatAn33nuv3fKEhAQYDAa8/vrrNc4n+eMl0c1p3bp1dpfU7969G7/99ps0ZyosLAx9+vTBqlWr7C57P3z4MH766SfccccdAAClUokxY8bghx9+qPGjFRo62tfYY0XUVLwUnKiZ3XjjjfDz88PkyZPx17/+FQqFAp9//rlTnRaaP38+fvrpJwwePBgzZsyA2WzGBx98gB49eiA5Oble26ioqMD//d//VWv39/fHzJkz8cYbb2Dq1KkYNmwYJkyYIF0KHhUVhaeeegqAdbTh1ltvxb333otu3bpBrVbj22+/RWZmJu677z4AwKpVq/Dhhx9i7Nix6NChAwoLC/HJJ5/AYDBIb9I1CQoKwpw5c7BgwQKMGDECd955J1JSUvDhhx9i4MCB1W7I2K9fP8TExODFF19EeXm53SkpADAYDFi6dCkeeOAB9OvXD/fddx+CgoKQlpaG9evXY/Dgwfjggw/q9bO7no0bN9Y44fzGG29E+/btpdcxMTEYMmQIZsyYgfLycrz33nsICAjAc889J/V56623MHLkSMTFxeGhhx6SLgU3Go2YP3++1O/111/HTz/9hGHDhmHatGno2rUr0tPT8fXXX2Pnzp3SJOH6aOyxImoy2a7TInJh17sUvHv37jX237Vrl7jhhhuEXq8X4eHh4rnnnhM//vijACC2bdsm9bvepeA1XRoNQMybN096fb1LwWfNmlVt3Xbt2onJkyfbtW3ZskX07dtXaDQa0aFDB/HPf/5TPP3000Kn013np3CV7VLnmh4dOnSQ+q1evVr07dtXaLVa4e/vLyZOnGh3CXN2draYNWuW6NKli/Dy8hJGo1HExsaKNWvWSH32798vJkyYINq2bSu0Wq0IDg4Wf/rTn8TevXvrrFMI66XfXbp0ER4eHiIkJETMmDFD5Obm1tj3xRdfFABETEzMdbe3bds2kZCQIIxGo9DpdKJDhw5iypQpdvXUdan8H9V2KTgAsWLFCiGE/e/HO++8IyIjI4VWqxVDhw4VBw8erLbdzZs3i8GDBwu9Xi8MBoMYPXq0OHr0aLV+qampYtKkSSIoKEhotVrRvn17MWvWLFFeXm5X3x8v8d62bZvd73RTjxVRYymEcKL/PhKRUxkzZgyOHDlS45wOkt+5c+cQHR2Nt956C88884zc5RA5Dc65ISIAQGlpqd3rkydPYsOGDS06MZaIyBE454aIAADt27fHlClTpHu+LF26FBqNxm7eBhGRK2C4ISIAwIgRI/Dll18iIyMDWq0WcXFxeP3112u8QRwRkTPjnBsiIiJyK5xzQ0RERG6F4YaIiIjcSqubc2OxWHDp0iX4+PhU+4RbIiIick5CCBQWFiI8PLzOz2lrdeHm0qVLiIyMlLsMIiIiaoTz58+jTZs2tfZpdeHGx8cHgPWHYzAYZK6GiIiI6qOgoACRkZHS+3htWl24sZ2KMhgMDDdEREQupj5TSjihmIiIiNwKww0RERG5FYYbIiIicisMN0RERORWGG6IiIjIrTDcEBERkVthuCEiIiK3wnBDREREboXhhoiIiNwKww0RERG5FYYbIiIicisMN0RERORWGG4cKKfYhJSMQrnLICIiatUYbhwk8Wgm+r2aiGfXHpS7FCIiolaN4cZBuocbAABHLhWguLxS5mqIiIhaL4YbBwn31SPcqIPZInDwfJ7c5RAREbVaDDcO1D/KHwCw51yuzJUQERG1Xgw3DjSgnR8AYG9qjsyVEBERtV4MNw50Q/sAAMDuszkoMXHeDRERkRwYbhyoU4g3Iv31KK+0YMeJbLnLISIiapUYbhxIoVDgtq6hAICfjmbIXA0REVHrxHDjYLd1CwEAbD2ehUqzReZqiIiIWh+GGwcbGOUHX08P5JVUYG8qr5oiIiJqaQw3DqZWKXFLl2AA1rsWExERUctiuGkGt1edmtp8LBNCCJmrISIial0YbprB0I5B0KiVSL1SglNZRXKXQ0RE1Kow3DQDL60agztY73mTeIynpoiIiFoSw00zibedmuK8GyIiohbFcNNMbu1iDTcHzufhcmG5zNUQERG1Hgw3zSTUqEOvNkYIAWw7niV3OURERK0Gw00ziu9qHb3hvBsiIqKWw3DTjGzh5ueTl1FWYZa5GiIiotaB4aYZdQ3zQbCPFmUVFuxP492KiYiIWgLDTTNSKBSIq7ok/NczOTJXQ0RE1Dow3DSzG9pXhZvTV2SuhIiIqHVguGlmcVXhJvl8HkpNnHdDRETU3Bhumlm7AE+EGXUwmTnvhoiIqCUw3DQzhUJx9dTUGZ6aIiIiam4MNy0gNtofALD3HEduiIiImhvDTQvo2cYIADhyKR9CCJmrISIicm8MNy2gY7APNColCsoqcT6nVO5yiIiI3BrDTQvQqJXoEuYDADh8KV/maoiIiNwbw00L6R5uPTV16CLDDRERUXOSNdzs2LEDo0ePRnh4OBQKBdatW1fnOtu3b0e/fv2g1WoRExODlStXNnudjtAjwgAAOMxwQ0RE1KxkDTfFxcXo3bs3lixZUq/+Z8+exahRo3DzzTcjOTkZTz75JB5++GH8+OOPzVxp0/WMsI7cHL7IScVERETNSS3nNx85ciRGjhxZ7/7Lli1DdHQ03nnnHQBA165dsXPnTrz77rtISEhorjIdolOID9RKBXJLKnApvwwRvnq5SyIiInJLLjXnJikpCfHx8XZtCQkJSEpKuu465eXlKCgosHvIQeehQqeQqknFPDVFRETUbFwq3GRkZCAkJMSuLSQkBAUFBSgtrfkS64ULF8JoNEqPyMjIlii1Rpx3Q0RE1PxcKtw0xpw5c5Cfny89zp8/L1stPa6Zd0NERETNQ9Y5Nw0VGhqKzMxMu7bMzEwYDAbo9TXPYdFqtdBqtS1RXp1s4ebQxQIIIaBQKGSuiIiIyP241MhNXFwctmzZYteWmJiIuLg4mSpqmK6hBigUQHZROXKKTXKXQ0RE5JZkDTdFRUVITk5GcnIyAOul3snJyUhLSwNgPaU0adIkqf/06dNx5swZPPfcczh+/Dg+/PBDrFmzBk899ZQc5TeYXqNCuNE6wnTuSrHM1RAREbknWcPN3r170bdvX/Tt2xcAMHv2bPTt2xdz584FAKSnp0tBBwCio6Oxfv16JCYmonfv3njnnXfwz3/+0+kvA79WVKAnAODMZYYbIiKi5iDrnJvhw4fXekO7mu4+PHz4cBw4cKAZq2peUQFe2HXqCkduiIiImolLzblxB9GBXgCAc9klMldCRETknhhuWlhUgDXcnM3myA0REVFzYLhpYdFBVSM3V4r5GVNERETNgOGmhUX6eUKpAEpMZmQVlstdDhERkdthuGlhGrUSbfysV0zx1BQREZHjMdzIIEqaVMxwQ0RE5GgMNzJoH8hJxURERM2F4UYGUQE8LUVERNRcGG5kIJ2W4o38iIiIHI7hRgbSjfyulMBi4eXgREREjsRwI4MIXz08VAqYKi24lF8qdzlERERuheFGBmqVEpH+1nk3/BgGIiIix2K4kUm07WMYOO+GiIjIoRhuZGKbVJzGcENERORQDDcyifDVAwAu5nHODRERkSMx3MgkXAo3ZTJXQkRE5F4YbmTSxq8q3ORy5IaIiMiRGG5kYjstlV1UjrIKs8zVEBERuQ+GG5n4enpA76ECAKTn89QUERGRozDcyEShUCCCp6aIiIgcjuFGRrZTU5d4xRQREZHDMNzIyHbF1AWGGyIiIodhuJERr5giIiJyPIYbGYX76gAA6fzwTCIiIodhuJFRiMEabjIKeLUUERGRozDcyCjMaD0tlZFfBiGEzNUQERG5B4YbGYVWjdyUmMwoLK+UuRoiIiL3wHAjI71GBaPeA4B19IaIiIiajuFGZmFG26RihhsiIiJHYLiRWWhVuMlkuCEiInIIhhuZ2ebdcOSGiIjIMRhuZGYbueHl4ERERI7BcCMz28hNBm/kR0RE5BAMNzK7OnJTLnMlRERE7oHhRmZXb+THkRsiIiJHYLiRme20VG5JBcoqzDJXQ0RE5PoYbmRm0Kuh87AehiyemiIiImoyhhuZKRQKfoAmERGRAzHcOAGGGyIiIsdhuHECtnk3WQw3RERETcZw4wRCDFoA/PBMIiIiR2C4cQI8LUVEROQ4DDdOwHYjP14tRURE1HQMN04glCM3REREDsNw4wSuPS0lhJC5GiIiItfGcOMEgqsmFJsqLcgrqZC5GiIiItfGcOMEtGoV/L00AIDMQp6aIiIiagqGGycR7MPLwYmIiByB4cZJ2K6YyuSkYiIioiZhuHEStiumMnk5OBERUZMw3DiJYF4OTkRE5BAMN05CGrnhnBsiIqImYbhxEqFG64RiXi1FRETUNLKHmyVLliAqKgo6nQ6xsbHYvXt3rf3fe+89dO7cGXq9HpGRkXjqqadQVub6gUC6kV8+59wQERE1hazhZvXq1Zg9ezbmzZuH/fv3o3fv3khISEBWVlaN/f/973/jhRdewLx583Ds2DF8+umnWL16Nf72t7+1cOWOZws3V4rLUWG2yFwNERGR65I13CxevBiPPPIIpk6dim7dumHZsmXw9PTE8uXLa+z/yy+/YPDgwbj//vsRFRWF22+/HRMmTKhztMcV+Htq4KFSQAggq5CjN0RERI0lW7gxmUzYt28f4uPjrxajVCI+Ph5JSUk1rnPjjTdi3759Upg5c+YMNmzYgDvuuOO636e8vBwFBQV2D2ekVCoQ7MN73RARETWVWq5vnJ2dDbPZjJCQELv2kJAQHD9+vMZ17r//fmRnZ2PIkCEQQqCyshLTp0+v9bTUwoULsWDBAofW3lxCDFpczCvlFVNERERNIPuE4obYvn07Xn/9dXz44YfYv38/vvnmG6xfvx6vvvrqddeZM2cO8vPzpcf58+dbsOKGsd2lmPe6ISIiajzZRm4CAwOhUqmQmZlp156ZmYnQ0NAa13n55ZfxwAMP4OGHHwYA9OzZE8XFxZg2bRpefPFFKJXVs5pWq4VWq3X8DjSDEN6lmIiIqMlkG7nRaDTo378/tmzZIrVZLBZs2bIFcXFxNa5TUlJSLcCoVCoAgBCi+YptIVfDDUduiIiIGku2kRsAmD17NiZPnowBAwZg0KBBeO+991BcXIypU6cCACZNmoSIiAgsXLgQADB69GgsXrwYffv2RWxsLE6dOoWXX34Zo0ePlkKOKwuV7nXDcENERNRYsoab8ePH4/Lly5g7dy4yMjLQp08fbNq0SZpknJaWZjdS89JLL0GhUOCll17CxYsXERQUhNGjR+O1116TaxcciiM3RERETacQ7nA+pwEKCgpgNBqRn58Pg8Egdzl2zmYX4+a3t8NTo8KRBQlQKBRyl0REROQUGvL+7VJXS7k722mpEpMZheWVMldDRETkmhhunIheo4JR7wGA826IiIgai+HGyXBSMRERUdMw3DgZ6UZ+DDdERESNwnDjZKSRG14xRURE1CgMN06GH8FARETUNAw3ToanpYiIiJqG4cbJMNwQERE1DcONk+GcGyIioqZhuHEyYVUjNznFJpRVmGWuhoiIyPUw3DgZo94DWrX1sGQVlMtcDRERkethuHEyCoWCV0wRERE1AcONE7LNu0nPL5W5EiIiItfDcOOEbPNu0nnFFBERUYMx3DihMF89ACA9jyM3REREDcVw44TCOXJDRETUaAw3TijMWDVyw3BDRETUYAw3TijMlxOKiYiIGovhxgnZRm6yi0wor+SN/IiIiBqC4cYJ+XlevZEfP2OKiIioYRhunJBCoUB41RVTl/IYboiIiBqC4cZJhXPeDRERUaMw3DipcKNt5IbhhoiIqCEYbpyU7bTURZ6WIiIiahCGGycVIYUbjtwQERE1BMONk7o6oZjhhoiIqCEYbpyUbULxpbxSCCFkroaIiMh1MNw4KdvITYnJjLySCpmrISIich0MN05K56FCoLcWAOfdEBERNQTDjRNr42cdvbmQWyJzJURERK6D4caJ2cLN+RyO3BAREdUXw40Ti/T3BMCRGyIiooZguHFi0shNLkduiIiI6ovhxolF+nHkhoiIqKEYbpzYtXNueK8bIiKi+mG4cWK2e92UVpiRU2ySuRoiIiLXwHDjxHQeKoQYrPe6ucB5N0RERPXCcOPk2lTNuznPeTdERET1wnDj5CKlG/lx5IaIiKg+GG6cnDRyk8ORGyIiovpguHFybQOs4Sb1CsMNERFRfTDcOLnoQC8AwNnsYpkrISIicg0MN04uKsAabi7ll6KswixzNURERM6P4cbJBXpr4KNVQwggjfNuiIiI6sRw4+QUCgWieGqKiIio3hhuXIAt3JxjuCEiIqoTw40L4KRiIiKi+mO4cQHRgdbLwRluiIiI6sZw4wJsV0ydu8JwQ0REVBeGGxdgOy2VWVCO4vJKmashIiJybgw3LsDXUwM/Tw8AHL0hIiKqC8ONi7h6xRTvdUNERFSbRoWb8+fP48KFC9Lr3bt348knn8THH3/ssMLIXjTn3RAREdVLo8LN/fffj23btgEAMjIycNttt2H37t148cUX8corrzi0QLLijfyIiIjqp1Hh5vDhwxg0aBAAYM2aNejRowd++eUXfPHFF1i5cmWDtrVkyRJERUVBp9MhNjYWu3fvrrV/Xl4eZs2ahbCwMGi1WnTq1AkbNmxozG64FNuk4jOXi2SuhIiIyLmpG7NSRUUFtFotAGDz5s248847AQBdunRBenp6vbezevVqzJ49G8uWLUNsbCzee+89JCQkICUlBcHBwdX6m0wm3HbbbQgODsbatWsRERGB1NRU+Pr6NmY3XErHEG8AwMmsIgghoFAoZK6IiIjIOTVq5KZ79+5YtmwZfv75ZyQmJmLEiBEAgEuXLiEgIKDe21m8eDEeeeQRTJ06Fd26dcOyZcvg6emJ5cuX19h/+fLlyMnJwbp16zB48GBERUVh2LBh6N27d2N2w6W0D/SGWqlAYVklMgrK5C6HiIjIaTUq3Lzxxhv46KOPMHz4cEyYMEEKF99//710uqouJpMJ+/btQ3x8/NVilErEx8cjKSmpxnW+//57xMXFYdasWQgJCUGPHj3w+uuvw2w2X/f7lJeXo6CgwO7hijRqpXRqKiWjUOZqiIiInFejTksNHz4c2dnZKCgogJ+fn9Q+bdo0eHp61msb2dnZMJvNCAkJsWsPCQnB8ePHa1znzJkz2Lp1KyZOnIgNGzbg1KlTmDlzJioqKjBv3rwa11m4cCEWLFhQzz1zbp1CfXAyqwgnMgsxvHP103ZERETUyJGb0tJSlJeXS8EmNTUV77333nXnyjiKxWJBcHAwPv74Y/Tv3x/jx4/Hiy++iGXLll13nTlz5iA/P196nD9/vtnqa26dQ3wAACkZnFRMRER0PY0aubnrrrswbtw4TJ8+HXl5eYiNjYWHhweys7OxePFizJgxo85tBAYGQqVSITMz0649MzMToaGhNa4TFhYGDw8PqFQqqa1r167IyMiAyWSCRqOpto5Wq5UmP7u6TrZwk+map9aIiIhaQqNGbvbv34+hQ4cCANauXYuQkBCkpqbis88+wz/+8Y96bUOj0aB///7YsmWL1GaxWLBlyxbExcXVuM7gwYNx6tQpWCwWqe3EiRMICwurMdi4m86h1nBzMrMIZouQuRoiIiLn1KhwU1JSAh8f6xvtTz/9hHHjxkGpVOKGG25Aampqvbcze/ZsfPLJJ1i1ahWOHTuGGTNmoLi4GFOnTgUATJo0CXPmzJH6z5gxAzk5OXjiiSdw4sQJrF+/Hq+//jpmzZrVmN1wOW39PaHzUKK80oK0HH4MAxERUU0adVoqJiYG69atw9ixY/Hjjz/iqaeeAgBkZWXBYDDUezvjx4/H5cuXMXfuXGRkZKBPnz7YtGmTNMk4LS0NSuXV/BUZGSl9v169eiEiIgJPPPEEnn/++cbshstRKRXoGOyDQxfzkZJRKF09RURERFcphBANPr+xdu1a3H///TCbzbjllluQmJgIwHpl0o4dO7Bx40aHF+ooBQUFMBqNyM/Pb1AQcxZPrzmI/+y/gNm3dcJfb+0odzlEREQtoiHv340aubnnnnswZMgQpKen291A79Zbb8XYsWMbs0mqp86h1jsVp2TyXjdEREQ1aVS4AYDQ0FCEhoZKnw7epk2bet/AjxrPdsXU8XReMUVERFSTRk0otlgseOWVV2A0GtGuXTu0a9cOvr6+ePXVV+2uZCLH6x5uBACcyS5GialS5mqIiIicT6NGbl588UV8+umnWLRoEQYPHgwA2LlzJ+bPn4+ysjK89tprDi2Srgry0SLEoEVmQTmOXirAgCh/uUsiIiJyKo0KN6tWrcI///lP6dPAAUhXL82cOZPhppn1jDAisyALhy7mM9wQERH9QaNOS+Xk5KBLly7V2rt06YKcnJwmF0W1s52aOnyR826IiIj+qFHhpnfv3vjggw+qtX/wwQfo1atXk4ui2vWMsIWbfJkrISIicj6NOi315ptvYtSoUdi8ebP0UQlJSUk4f/48NmzY4NACqbqebazh5mRWIUpNZug1qjrWICIiaj0aNXIzbNgwnDhxAmPHjkVeXh7y8vIwbtw4HDlyBJ9//rmja6Q/CPbRItBbC4sAjqZz9IaIiOhajbpD8fUcPHgQ/fr1g9lsdtQmHc7V71Bs8/CqPdh8LAtz/9QNDw6JlrscIiKiZtWQ9+9GjdyQ/PpE+gIADpzPk7UOIiIiZ8Nw46L6tfUDAOxPzZW5EiIiIufCcOOiekf6QqkALuaVIrOgTO5yiIiInEaDrpYaN25crcvz8vKaUgs1gJdWjc6hBhxLL8D+1FyM7Bkmd0lEREROoUHhxmg01rl80qRJTSqI6q9/O18cSy/AXoYbIiIiSYPCzYoVK5qrDmqEgVH++Nevadh9lneFJiIisuGcGxc2KNr6uVJHLuWjsKxC5mqIiIicA8ONCwsz6tHW3xMWAezjVVNEREQAGG5cXmzV6M1vPDVFREQEgOHG5dlOTXHeDRERkRXDjYu7oX0AAOD3C3koMVXKXA0REZH8GG5cXBs/PSJ89agwC47eEBERgeHG5SkUCgyJCQQA7DqVLXM1RERE8mO4cQM3xlhPTe08dUXmSoiIiOTHcOMGBleN3BxLL8DlwnKZqyEiIpIXw40bCPTWonu4AQDw88nLMldDREQkL4YbNzG8cxAAYHsKww0REbVuDDduYlinYADWkRuzRchcDRERkXwYbtxEv7a+8NGpkVtSgeTzeXKXQ0REJBuGGzehVikxrJP11NTW45kyV0NERCQfhhs3cmtX66mpzUezZK6EiIhIPgw3bmR4p2ColAqkZBYi7UqJ3OUQERHJguHGjfh5aTAwyg8A8NPRDJmrISIikgfDjZtJ6B4KANh4mOGGiIhaJ4YbNzOyRxgAYF9qLtLzS2WuhoiIqOUx3LiZUKMOA9pZT01tPMTRGyIian0YbtzQHT2tozcbDqXLXAkREVHLY7hxQyN7Wufd7E3NxaU8npoiIqLWheHGDYUZ9RgU5Q8A+P7gJZmrISIialkMN25qTN8IAMC6AxdlroSIiKhlMdy4qVE9w6BRKXE8oxDH0gvkLoeIiKjFMNy4KaOnB27pYv04hm85ekNERK0Iw40bu/bUlNkiZK6GiIioZTDcuLGbuwTBz9MDWYXl2HHistzlEBERtQiGGzemVaswtm8bAMDqPedlroaIiKhlMNy4ufEDIwEAm49l4nJhuczVEBERNT+GGzfXOdQHfSJ9UWkRWLvvgtzlEBERNTuGm1bg/kFtAQBf7k6DhROLiYjIzTHctAJ/6h0GH60aaTkl2HU6W+5yiIiImhXDTSvgqVFjXD/rZeGfJ6XKXA0REVHzYrhpJR6IawfAOrH4Qm6JzNUQERE1H4abViIm2AeDYwJgEcC/fk2TuxwiIqJmw3DTikyKiwIAfLUnDaUms7zFEBERNROnCDdLlixBVFQUdDodYmNjsXv37nqt99VXX0GhUGDMmDHNW6CbiO8agkh/PfJKKvCf/bwsnIiI3JPs4Wb16tWYPXs25s2bh/3796N3795ISEhAVlZWreudO3cOzzzzDIYOHdpClbo+lVKBKTdGAwCW7zzLy8KJiMgtyR5uFi9ejEceeQRTp05Ft27dsGzZMnh6emL58uXXXcdsNmPixIlYsGAB2rdv34LVur7xAyPho1XjTHYxth6vPUASERG5IlnDjclkwr59+xAfHy+1KZVKxMfHIykp6brrvfLKKwgODsZDDz3UEmW6FW+tGvfHWm/q99GO0zJXQ0RE5Hiyhpvs7GyYzWaEhITYtYeEhCAjI6PGdXbu3IlPP/0Un3zySb2+R3l5OQoKCuwerd3UwdHwUCmw51wu9qXmyl0OERGRQ8l+WqohCgsL8cADD+CTTz5BYGBgvdZZuHAhjEaj9IiMjGzmKp1fqFGHsX2tN/Vbuv2UzNUQERE5lqzhJjAwECqVCpmZmXbtmZmZCA0Nrdb/9OnTOHfuHEaPHg21Wg21Wo3PPvsM33//PdRqNU6frn6aZc6cOcjPz5ce58+fb7b9cSXTh3WAQgFsPpaF4xkczSIiIvcha7jRaDTo378/tmzZIrVZLBZs2bIFcXFx1fp36dIFhw4dQnJysvS48847cfPNNyM5ObnGURmtVguDwWD3IKB9kDfu6BkGAPhgK0dviIjIfajlLmD27NmYPHkyBgwYgEGDBuG9995DcXExpk6dCgCYNGkSIiIisHDhQuh0OvTo0cNufV9fXwCo1k51mzU8But/T8f6Q+l4MqsIMcHecpdERETUZLKHm/Hjx+Py5cuYO3cuMjIy0KdPH2zatEmaZJyWlgal0qWmBrmMbuEG3NYtBIlHM7Fk2ym8O76P3CURERE1mUII0aru5FZQUACj0Yj8/HyeogJw+GI+/vT+TigVQOLsYegQxNEbIiJyPg15/+aQSCvXI8KI+K4hsAjg3cQTcpdDRETUZAw3hNm3dQIA/Pf3dBxL55VTRETk2hhuCN3CDRjVy3rl1Ds/cfSGiIhcG8MNAQCeiu8EpQLYfCwT+9N412IiInJdDDcEAIgJ9sbd/doAABZtPI5WNs+ciIjcCMMNSZ66rRM0aiV2n83BlmP8xHAiInJNDDckCffV48HB0QCANzYdR6XZInNFREREDcdwQ3ZmDO8AX08PnMwqwtp9F+Quh4iIqMEYbsiOUe+Bx2/pCABYnHgCJaZKmSsiIiJqGIYbquYvN7RFpL8eWYXl+Oh/Z+Quh4iIqEEYbqgarVqFF0Z0BQB8tOM0LuWVylwRERFR/THcUI3u6BmKQVH+KKuwYNHG43KXQ0REVG8MN1QjhUKBuaO7QaEAvj94CXvP5chdEhERUb0w3NB19YgwYvyASADA/B+OwGzhjf2IiMj5MdxQrZ5J6AwfnRqHLxbg37+lyl0OERFRnRhuqFaB3lo8c3tnAMA7iSeQU2ySuSIiIqLaMdxQnSbGtkWXUB/klVTgDU4uJiIiJ8dwQ3VSq5T4vzE9AACr957HHk4uJiIiJ8ZwQ/UyIMof9w20Ti5+/j+/o6zCLHNFRERENWO4oXqbM7IrQgxanLlcjLd/TJG7HCIiohox3FC9GT09sGhcLwDAp7vOYvdZnp4iIiLnw3BDDXJzl2DcO6ANhACeXXuQH6xJREROh+GGGuylP3VDmFGH1CslvHqKiIicDsMNNZhB54E37raenlqVlIpfTmfLXBEREdFVDDfUKDd1CsL9sW0BAM+t/R1F5Tw9RUREzoHhhhrtb3d0RRs/PS7kluL1DcfkLoeIiAgAww01gbdWjTfvsZ6e+vdvadhx4rLMFRERETHcUBPd2CEQk+PaAbDe3K+grELmioiIqLVjuKEme35kF7QL8ER6fhle/eGo3OUQEVErx3BDTeapUeOte3pDoQC+3ncBW49nyl0SERG1Ygw35BCDov3x4OBoAMAL/zmEnGKTzBUREVFrxXBDDvNsQme0D/JCVmE5nl6TDItFyF0SERG1Qgw35DA6DxWW3N8PWrUS21Iu48Ptp+QuiYiIWiGGG3KormEGvDqmBwDgncQTvDyciIhaHMMNOdy9AyIxYVAkhAD++tUBXMgtkbskIiJqRRhuqFnMG90dvdoYkVdSgZlf7EdZhVnukoiIqJVguKFmofNQ4cOJ/eDn6YHfL+Rj/vdH5C6JiIhaCYYbajZt/Dzx9/v6QqEAvtpzHp8nnZO7JCIiagUYbqhZ3dQpCM8ldAEAzP/hKHaezJa5IiIicncMN9Tspg9rj7F9I2C2CMz4Yh9OZhbKXRIREbkxhhtqdgqFAgvH9cSAdn4oLKvE1JV7kFVYJndZRETkphhuqEXoPFT46IH+aBfgiQu5pXho5V4U8hPEiYioGTDcUIsJ8NZi5dRB8PfS4NDFfDy0ai9KTbxEnIiIHIvhhlpUdKAXPntwEHy0auw+m4Pp/9qH8koGHCIichyGG2pxPSKMWPngQOg9VPjfict44stkVJotcpdFRERuguGGZNG/nT8+mTQAGpUSm45k4Lm1v/NTxImIyCEYbkg2QzoG4oP7+0KlVOCbAxcx7/sjEIIBh4iImobhhmR1e/dQLL63NxQK4PNfU/HGphQGHCIiahKGG5LdXX0i8NqYngCAZf87jcWJJxhwiIio0RhuyCncH9sWL43qCgB4f+spLNp0nAGHiIgaheGGnMbDQ9tj7p+6AQA++t8ZPP31QZgqeRUVERE1DMMNOZUHh0Rj4bie1knG+y9i8vLdyC/lnYyJiKj+GG7I6UwY1Bb/nDwAXhoVks5cwT1Lf8H5nBK5yyIiIhfBcENO6ebOwVgzPQ4hBi1OZhVh7Ie/4PcLeXKXRURELsApws2SJUsQFRUFnU6H2NhY7N69+7p9P/nkEwwdOhR+fn7w8/NDfHx8rf3JdXUPN2LdrMHoEuqD7KJyjP/oVyQezZS7LCIicnKyh5vVq1dj9uzZmDdvHvbv34/evXsjISEBWVlZNfbfvn07JkyYgG3btiEpKQmRkZG4/fbbcfHixRaunFpCmFGPr6fH4aZOQSitMGPa53uxctdZucsiIiInphAyX28bGxuLgQMH4oMPPgAAWCwWREZG4vHHH8cLL7xQ5/pmsxl+fn744IMPMGnSpDr7FxQUwGg0Ij8/HwaDocn1U8uoMFsw97sj+HJ3GgDgLze0xdw/dYdGLXs+JyKiFtCQ929Z3xlMJhP27duH+Ph4qU2pVCI+Ph5JSUn12kZJSQkqKirg7+9f4/Ly8nIUFBTYPcj1eKiUeH1sDzw/ogsA4F+/puG+j5NwIZcTjYmIyJ6s4SY7OxtmsxkhISF27SEhIcjIyKjXNp5//nmEh4fbBaRrLVy4EEajUXpERkY2uW6Sh0KhwIzhHbB8ygD46NTYn5aHke/9jP/su8Ab/hERkcSlx/QXLVqEr776Ct9++y10Ol2NfebMmYP8/Hzpcf78+Raukhztli4h2PDXoejX1heF5ZV4+uuDmPnFfuQUm+QujYiInICs4SYwMBAqlQqZmfZXwGRmZiI0NLTWdd9++20sWrQIP/30E3r16nXdflqtFgaDwe5Bri/S3xNrHo3DswmdoVYqsPFwBhLe24FtKTVPRCciotZD1nCj0WjQv39/bNmyRWqzWCzYsmUL4uLirrvem2++iVdffRWbNm3CgAEDWqJUckJqlRKzbo7BulmDERPsjcuF5Zi6Yg9e/PYQSkyVcpdHREQykf201OzZs/HJJ59g1apVOHbsGGbMmIHi4mJMnToVADBp0iTMmTNH6v/GG2/g5ZdfxvLlyxEVFYWMjAxkZGSgqKhIrl0gmfWIMOK/jw/Bg4OjAQBf/JaGO/7+M/an5cpcGRERyUH2cDN+/Hi8/fbbmDt3Lvr06YPk5GRs2rRJmmSclpaG9PR0qf/SpUthMplwzz33ICwsTHq8/fbbcu0COQGdhwpzR3fDFw/HIsyow7krJbhn6S9468fjKKswy10eERG1INnvc9PSeJ8b95dfWoH53x/BtwesN3aM9Ndj/ujuuLVrSB1rEhGRs3KZ+9wQNQej3gPvju+DZX/ph1CDDudzSvHQqr14eNVepF4plrs8IiJqZhy5IbdWXF6Jf2w9iU9/PotKi4BGpcSUwVGYdXMMjHoPucsjIqJ6asj7N8MNtQonMwvxyn+P4ueT2QAAfy8Nnri1IyYMasuPcCAicgEMN7VguGm9hBDYnnIZ/7f+KE5ftp6eivTX46+3dMTYvhFQqxhyiIicFcNNLRhuqMJsweo95/H3LSdxubAcABAd6IVZN8dgTJ9whhwiIifEcFMLhhuyKTFV4rOkVHz0v9PILakAALT198TM4R0wrl8bnq4iInIiDDe1YLihPyoqr8TnSan4589ncKXq86nCjDpMHRyF+wa1hUHHicdERHJjuKkFww1dT4mpEv/+LQ0f7Tgjna7y1qrx5wFtMDkuClGBXjJXSETUejHc1ILhhupSXmnGugMX8c+fz+JklvVjPRQKYHinIEy+MQo3dQyCUqmQuUoiotaF4aYWDDdUX0II7DiZjRW7zmJ7ymWpva2/J+7p3wZ392+DCF+9jBUSEbUeDDe1YLihxjibXYzPks5h7b4LKCyzfuK4QgEMiQnEnwdE4vZuIdB5qGSukojIfTHc1ILhhpqixFSJTYcz8PXeC0g6c0VqN+jUGNkjDCN7huLGDoG80oqIyMEYbmrBcEOOcj6nBF/vu4D/7LuAi3mlUrtBp8Zt3UIxskcohnQM5IgOEZEDMNzUguGGHM1iEfj17BVsOJSOTYczkV1ULi3z1qpxa9dgjOwRhuGdgxh0iIgaieGmFgw31JzMFoF9qblVQScDGQVl0jJPjQo3dw7GyJ6hGNYpCD68fw4RUb0x3NSC4YZaisUicOB8HjYdTseGQxl2p648VAr0b+eHoR2DMKxTELqFGXh5ORFRLRhuasFwQ3IQQuDQxXxsOJSBH49k4Gx2sd3yAC8NhnQMxNCOQRgcE4AwIy8xJyK6FsNNLRhuSG5CCKReKcGOk5ex40Q2kk5no9hktusTHeiFG9oH4Ib2/oiNDkCoUSdTtUREzoHhphYMN+RsKswW7E/Nxc8ns/Hzycs4dDEflj/8q4z012NglD8GRvmjX1s/xAR7Q8XTWETUijDc1ILhhpxdfmkF9pzNQdKZK9h9NgdHLlUPOz46NfpE+qJPpC/6tvVFzwhfBPlo5SmYiKgFMNzUguGGXE1BWQX2peZif2ou9pzLwaEL+dVOYwFAiEGLnhFG9IgwomfVI9jA01lE5B4YbmrBcEOurtJswbH0QiRfyENyWh4OXsjD6ctFqOlfcpCPNfB0DfNB1zADuoT6ICrAC2oV76BMRK6F4aYWDDfkjorLK3E0vQCHLuTj8MV8HLqYj9OXi6qdzgIAjUqJ9kFe6BLqg44hPugU4oOYYG9E+ukZeojIaTHc1ILhhlqLElMljqUX4PDFAhzPKMCx9EKcyCxESQ2ntABr6IkO9EKHYC90CPJGhyBvtA/yQlSgFwy84SARyYzhphYMN9SaWSwCF/NKkZJRiJRMa9g5mVmE05eLUF5pue56AV4atAvwRLsAr6qv1udRAV7w8/SAQsErt4ioeTHc1ILhhqg6W+g5lWUNOtZHMc5mF+NyYXmt6/po1Yj090Skvx5t/DzRxk+PCF/r8wg/PYx6jvoQUdMx3NSC4YaoYQrLKpB6pcT6yClGanYJzl0pRlpOCdLzy+pc30erRrivHhF+eoQadQgz6BDmq0eoQYdQo/XhrVW3wJ4QkStjuKkFww2R45SazLiQW4K0nBKczynBhdxSXMwrxYXcUlzILUFuSUW9tuOjUyPYR4swox7BPloEG3QI9tEixKBDkI8WIQYtgny08NQwBBG1Vg15/+ZfCiJqNL1GhY4h1quualJiqsSlvDJczCvFpbxSpOeXISPf9rUM6fllKCqvRGGZ9XH6cnGN27Hx0qgQ5GMNOoHeWgR4axDgpUWgtwaB3lr4e2kQ6KNFgJcGRj3nAhG1Vgw3RNRsPDVqxAR7IybY+7p9isorkZFfiqyCcmQUlCGzoByZBWW4XFiOrMIyZBVaX5dVWFBsMqP4SgnOXSmp83urlAr4eWoQ4KWBr6cH/Dw18PPSwO8Pz309NfCvem7QefDT2YncAMMNEcnKW6tGTLAPYoJrHv0BrB82WlReicuF5cguMlV9tT6uFJuQXWj9mlP1vLC8EmaLkPrUl1IBGPQeMP7hcW2bQWe/zLcqFHnr1Py8LyInwXBDRE5PoVDAR+cBH50H2gfV3b+80oycYhOuFJmQW2JCbkkF8kpMyC2uqHp9tS2n2IS8kgoUlVfCIoC8kgrk1XOu0B95alTw0anhrVVX1auGoSoQ+ejU8NGq4am1fvXRqeGltT68tWp469Tw1qjhpVXxZopETcRwQ0RuR6tWIcyoR5hRX+91TJUW5JaYkF9agYJSa8DJL7U+CsqueV5aiYLSq6/zSk0oq7DeI6jEZEaJyYxM1H+0qCY6DyW8tWp4amzhRyUFIS+NCp4aNTw11ja9hwreWjX0GhU8NSroNSp4aayv9R4qqV2nVvGUG7UaDDdERAA0aiVCDDqENOLDRk2VFhSWVaCwrBJF5ZUoqHpufVgDUbHJGoqKyq19iqr6Fpusz4vLzTCZrSGprMKCsgoTAJND91HnoawWfHQeVc//8FrnoYTeQwWthxJ6jVrqo/NQQqtWQa9RQudh7a9RKa0BykMFrVrJkSeSHcMNEVETadRKBHhrEeCtbdJ2TJUWFNvCT3klSkyVKCo3o8SuzYwSkzUMlZrMKDJVorSqrdRkRrHJfPV1hVkaVQKuCU21X5TWZCqlAlq1Ugo7tq9atRIau9dVX6sCk9ZDCa3qah+NWglN1WtNVX8PlaLquRIaVdU6Vcuv7euhVHKkqhVjuCEichLWN2brlVyOYrEIlFaYrQ+TGcVVIcgafMwoNVmk5WVV7SUm67LySms4sgYlC8pM1rbySotdvwqzxS5EmS1COkUnJw+VAh4qpRSk1CoFNColPKpCkG259bk1HHnY2pVKqFUK6Dzs17M+FNJztUohvVYrldCo7Z+rlfbr2LalUiqgrmq39lHw1gUOxHBDROTGlEqFNF+nOZktwhp8KiwwmS1V4ciC8goLyiqtIchUaYGp0oLyymuWV61zbZup0rqNa7dnW89UaUGFuWp5he2r9ZRehdn+nrQVZoEKsy1kNW6SeEtSKxXWsFQVrNQqJTyqQpBde1WbSqmASqGwBjKlNRxJ26habgtPGvXVdVVKZdVX68P23BrKFFBe06ZWKq/2sa2vqFqmst/WtdvUe6gQ3IhTvA77Wcr2nYmIyG2olIqqic7y1WCxCGsQMltQUWn7KlBWaZZCUYVZ2AWkCrMFlVVtthBVYbag0iJQaRYoqaiE2Syq+ld9rbSg0mKBqVKg0mKB2SKkbV/7tdK2PdvySut2K8wWWGr4bIBKi0ClRaAM1/8QW1fRt60vvp05WLbvz3BDRERuQalUQKe0Tmx2dhaLQIXFGqwqzVefXw1W1pBUabkawGzByBqmrMsswjpXy1z1vPKaYHZ1O9ZRL3NVeLJ9tdVglvpbn9fUr9Ku3XL1tbnmdp1a3mPAcENERNTClEoFtEoV+JmxzYPX6xEREZFbYbghIiIit8JwQ0RERG6F4YaIiIjcCsMNERERuRWGGyIiInIrDDdERETkVhhuiIiIyK0w3BAREZFbYbghIiIit8JwQ0RERG6F4YaIiIjcCsMNERERuRWGGyIiInIrre7D1oUQAICCggKZKyEiIqL6sr1v297Ha9Pqwk1hYSEAIDIyUuZKiIiIqKEKCwthNBpr7aMQ9YlAbsRiseDSpUvw8fGBQqFw6LYLCgoQGRmJ8+fPw2AwOHTbzsDd9w9w/33k/rk+d99H7p/ra659FEKgsLAQ4eHhUCprn1XT6kZulEol2rRp06zfw2AwuO0vLeD++we4/z5y/1yfu+8j98/1Ncc+1jViY8MJxURERORWGG6IiIjIrTDcOJBWq8W8efOg1WrlLqVZuPv+Ae6/j9w/1+fu+8j9c33OsI+tbkIxERERuTeO3BAREZFbYbghIiIit8JwQ0RERG6F4YaIiIjcCsONgyxZsgRRUVHQ6XSIjY3F7t275S6pRgsXLsTAgQPh4+OD4OBgjBkzBikpKXZ9hg8fDoVCYfeYPn26XZ+0tDSMGjUKnp6eCA4OxrPPPovKykq7Ptu3b0e/fv2g1WoRExODlStXNvfuYf78+dVq79Kli7S8rKwMs2bNQkBAALy9vXH33XcjMzPTJfbNJioqqto+KhQKzJo1C4DrHb8dO3Zg9OjRCA8Ph0KhwLp16+yWCyEwd+5chIWFQa/XIz4+HidPnrTrk5OTg4kTJ8JgMMDX1xcPPfQQioqK7Pr8/vvvGDp0KHQ6HSIjI/Hmm29Wq+Xrr79Gly5doNPp0LNnT2zYsKFZ96+iogLPP/88evbsCS8vL4SHh2PSpEm4dOmS3TZqOuaLFi1yiv2rax8BYMqUKdXqHzFihF0fVz2GAGr896hQKPDWW29JfZz5GNbnfaEl/3Y65P1UUJN99dVXQqPRiOXLl4sjR46IRx55RPj6+orMzEy5S6smISFBrFixQhw+fFgkJyeLO+64Q7Rt21YUFRVJfYYNGyYeeeQRkZ6eLj3y8/Ol5ZWVlaJHjx4iPj5eHDhwQGzYsEEEBgaKOXPmSH3OnDkjPD09xezZs8XRo0fF+++/L1Qqldi0aVOz7t+8efNE9+7d7Wq/fPmytHz69OkiMjJSbNmyRezdu1fccMMN4sYbb3SJfbPJysqy27/ExEQBQGzbtk0I4XrHb8OGDeLFF18U33zzjQAgvv32W7vlixYtEkajUaxbt04cPHhQ3HnnnSI6OlqUlpZKfUaMGCF69+4tfv31V/Hzzz+LmJgYMWHCBGl5fn6+CAkJERMnThSHDx8WX375pdDr9eKjjz6S+uzatUuoVCrx5ptviqNHj4qXXnpJeHh4iEOHDjXb/uXl5Yn4+HixevVqcfz4cZGUlCQGDRok+vfvb7eNdu3aiVdeecXumF77b1bO/atrH4UQYvLkyWLEiBF29efk5Nj1cdVjKISw26/09HSxfPlyoVAoxOnTp6U+znwM6/O+0FJ/Ox31fspw4wCDBg0Ss2bNkl6bzWYRHh4uFi5cKGNV9ZOVlSUAiP/9739S27Bhw8QTTzxx3XU2bNgglEqlyMjIkNqWLl0qDAaDKC8vF0II8dxzz4nu3bvbrTd+/HiRkJDg2B34g3nz5onevXvXuCwvL094eHiIr7/+Wmo7duyYACCSkpKEEM69b9fzxBNPiA4dOgiLxSKEcO3j98c3DovFIkJDQ8Vbb70lteXl5QmtViu+/PJLIYQQR48eFQDEnj17pD4bN24UCoVCXLx4UQghxIcffij8/Pyk/RNCiOeff1507txZen3vvfeKUaNG2dUTGxsrHn300Wbbv5rs3r1bABCpqalSW7t27cS777573XWcZf+EqHkfJ0+eLO66667rruNux/Cuu+4St9xyi12bKx3DP74vtOTfTke9n/K0VBOZTCbs27cP8fHxUptSqUR8fDySkpJkrKx+8vPzAQD+/v527V988QUCAwPRo0cPzJkzByUlJdKypKQk9OzZEyEhIVJbQkICCgoKcOTIEanPtT8TW5+W+JmcPHkS4eHhaN++PSZOnIi0tDQAwL59+1BRUWFXV5cuXdC2bVupLmfftz8ymUz417/+hQcffNDug2Bd+fhd6+zZs8jIyLCrxWg0IjY21u6Y+fr6YsCAAVKf+Ph4KJVK/Pbbb1Kfm266CRqNRuqTkJCAlJQU5ObmSn2cYZ/z8/OhUCjg6+tr175o0SIEBASgb9++eOutt+yG+11h/7Zv347g4GB07twZM2bMwJUrV+zqd5djmJmZifXr1+Ohhx6qtsxVjuEf3xda6m+nI99PW90HZzpadnY2zGaz3QEFgJCQEBw/flymqurHYrHgySefxODBg9GjRw+p/f7770e7du0QHh6O33//Hc8//zxSUlLwzTffAAAyMjJq3F/bstr6FBQUoLS0FHq9vln2KTY2FitXrkTnzp2Rnp6OBQsWYOjQoTh8+DAyMjKg0WiqvWmEhITUWbcz7FtN1q1bh7y8PEyZMkVqc+Xj90e2emqq5dpag4OD7Zar1Wr4+/vb9YmOjq62DdsyPz+/6+6zbRstoaysDM8//zwmTJhg94GDf/3rX9GvXz/4+/vjl19+wZw5c5Ceno7FixdL++DM+zdixAiMGzcO0dHROH36NP72t79h5MiRSEpKgkqlcqtjuGrVKvj4+GDcuHF27a5yDGt6X2ipv525ubkOez9luGnFZs2ahcOHD2Pnzp127dOmTZOe9+zZE2FhYbj11ltx+vRpdOjQoaXLbJCRI0dKz3v16oXY2Fi0a9cOa9asadHQ0VI+/fRTjBw5EuHh4VKbKx+/1qyiogL33nsvhBBYunSp3bLZs2dLz3v16gWNRoNHH30UCxcudInb+N93333S8549e6JXr17o0KEDtm/fjltvvVXGyhxv+fLlmDhxInQ6nV27qxzD670vuBqelmqiwMBAqFSqarPGMzMzERoaKlNVdXvsscfw3//+F9u2bUObNm1q7RsbGwsAOHXqFAAgNDS0xv21Lautj8FgaNGQ4evri06dOuHUqVMIDQ2FyWRCXl5etbrqqtu2rLY+Lb1vqamp2Lx5Mx5++OFa+7ny8bPVU9u/r9DQUGRlZdktr6ysRE5OjkOOa0v8O7YFm9TUVCQmJtqN2tQkNjYWlZWVOHfuHADn378/at++PQIDA+1+J139GALAzz//jJSUlDr/TQLOeQyv977QUn87Hfl+ynDTRBqNBv3798eWLVukNovFgi1btiAuLk7GymomhMBjjz2Gb7/9Flu3bq02DFqT5ORkAEBYWBgAIC4uDocOHbL7Y2T7g9ytWzepz7U/E1uflv6ZFBUV4fTp0wgLC0P//v3h4eFhV1dKSgrS0tKkulxp31asWIHg4GCMGjWq1n6ufPyio6MRGhpqV0tBQQF+++03u2OWl5eHffv2SX22bt0Ki8UiBbu4uDjs2LEDFRUVUp/ExER07twZfn5+Uh859tkWbE6ePInNmzcjICCgznWSk5OhVCqlUznOvH81uXDhAq5cuWL3O+nKx9Dm008/Rf/+/dG7d+86+zrTMazrfaGl/nY69P20QdOPqUZfffWV0Gq1YuXKleLo0aNi2rRpwtfX127WuLOYMWOGMBqNYvv27XaXJJaUlAghhDh16pR45ZVXxN69e8XZs2fFd999J9q3by9uuukmaRu2S/5uv/12kZycLDZt2iSCgoJqvOTv2WefFceOHRNLlixpkculn376abF9+3Zx9uxZsWvXLhEfHy8CAwNFVlaWEMJ6OWPbtm3F1q1bxd69e0VcXJyIi4tziX27ltlsFm3bthXPP/+8XbsrHr/CwkJx4MABceDAAQFALF68WBw4cEC6WmjRokXC19dXfPfdd+L3338Xd911V42Xgvft21f89ttvYufOnaJjx452lxHn5eWJkJAQ8cADD4jDhw+Lr776Snh6ela7zFatVou3335bHDt2TMybN88hl9nWtn8mk0nceeedok2bNiI5Odnu36TtCpNffvlFvPvuuyI5OVmcPn1a/Otf/xJBQUFi0qRJTrF/de1jYWGheOaZZ0RSUpI4e/as2Lx5s+jXr5/o2LGjKCsrk7bhqsfQJj8/X3h6eoqlS5dWW9/Zj2Fd7wtCtNzfTke9nzLcOMj7778v2rZtKzQajRg0aJD49ddf5S6pRgBqfKxYsUIIIURaWpq46aabhL+/v9BqtSImJkY8++yzdvdJEUKIc+fOiZEjRwq9Xi8CAwPF008/LSoqKuz6bNu2TfTp00doNBrRvn176Xs0p/Hjx4uwsDCh0WhERESEGD9+vDh16pS0vLS0VMycOVP4+fkJT09PMXbsWJGenu4S+3atH3/8UQAQKSkpdu2uePy2bdtW4+/k5MmThRDWy8FffvllERISIrRarbj11lur7feVK1fEhAkThLe3tzAYDGLq1KmisLDQrs/BgwfFkCFDhFarFREREWLRokXValmzZo3o1KmT0Gg0onv37mL9+vXNun9nz5697r9J232L9u3bJ2JjY4XRaBQ6nU507dpVvP7663bBQM79q2sfS0pKxO233y6CgoKEh4eHaNeunXjkkUeqvVm56jG0+eijj4Rerxd5eXnV1nf2Y1jX+4IQLfu30xHvp4qqHSMiIiJyC5xzQ0RERG6F4YaIiIjcCsMNERERuRWGGyIiInIrDDdERETkVhhuiIiIyK0w3BAREZFbYbgholZPoVBg3bp1cpdBRA7CcENEspoyZQoUCkW1x4gRI+QujYhclFruAoiIRowYgRUrVti1abVamaohIlfHkRsikp1Wq0VoaKjdw/ZJyAqFAkuXLsXIkSOh1+vRvn17rF271m79Q4cO4ZZbboFer0dAQACmTZuGoqIiuz7Lly9H9+7dodVqERYWhscee8xueXZ2NsaOHQtPT0907NgR33//ffPuNBE1G4YbInJ6L7/8Mu6++24cPHgQEydOxH333Ydjx44BAIqLi5GQkAA/Pz/s2bMHX3/9NTZv3mwXXpYuXYpZs2Zh2rRpOHToEL7//nvExMTYfY8FCxbg3nvvxe+//4477rgDEydORE5OTovuJxE5SIM/apOIyIEmT54sVCqV8PLysnu89tprQgjrJxZPnz7dbp3Y2FgxY8YMIYQQH3/8sfDz8xNFRUXS8vXr1wulUil98nR4eLh48cUXr1sDAPHSSy9Jr4uKigQAsXHjRoftJxG1HM65ISLZ3XzzzVi6dKldm7+/v/Q8Li7ObllcXBySk5MBAMeOHUPv3r3h5eUlLR88eDAsFgtSUlKgUChw6dIl3HrrrbXW0KtXL+m5l5cXDAYDsrKyGrtLRCQjhhsikp2Xl1e100SOotfr69XPw8PD7rVCoYDFYmmOkoiomXHODRE5vV9//bXa665duwIAunbtioMHD6K4uFhavmvXLiiVSnTu3Bk+Pj6IiorCli1bWrRmIpIPR26ISHbl5eXIyMiwa1Or1QgMDAQAfP311xgwYACGDBmCL774Art378ann34KAJg4cSLmzZuHyZMnY/78+bh8+TIef/xxPPDAAwgJCQEAzJ8/H9OnT0dwcDBGjhyJwsJC7Nq1C48//njL7igRtQiGGyKS3aZNmxAWFmbX1rlzZxw/fhyA9Uqmr776CjNnzkRYWBi+/PJLdOvWDQDg6emJH3/8EU888QQGDhwIT09P3H333Vi8eLG0rcmTJ6OsrAzvvvsunnnmGQQGBuKee+5puR0kohalEEIIuYsgIroehUKBb7/9FmPGjJG7FCJyEZxzQ0RERG6F4YaIiIjcCufcEJFT45lzImoojtwQERGRW2G4ISIiIrfCcENERERuheGGiIiI3ArDDREREbkVhhsiIiJyKww3RERE5FYYboiIiMitMNwQERGRW/l/AdpnVhNjchcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    # 前向传播\n",
    "    outputs = model(X_train)\n",
    "    loss = criterion(outputs, y_train)\n",
    "\n",
    "    # 反向传播和优化\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 记录损失值\n",
    "    losses.append(loss.item())\n",
    "\n",
    "    # 打印训练信息\n",
    "    if (epoch + 1) % 100 == 0:\n",
    "        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')\n",
    "\n",
    "time_all = time.time() - start_time\n",
    "print(f'Training time: {time_all:.2f} seconds')\n",
    "\n",
    "# 可视化损失曲线\n",
    "plt.plot(range(num_epochs), losses)\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.title('Training Loss over Epochs')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc986a30",
   "metadata": {},
   "source": [
    "你这时可能会好奇，不是说gpu比cpu快很多吗？怎么cpu跑了3s，gpu跑了11s。\n",
    "\n",
    "你问AI，他会告诉你，对于非常小的数据集和简单的模型，CPU 通常会比 GPU 更快。实际上，这并非本质的原因。\n",
    "\n",
    "这需要你进一步理解二者的区别，深度学习项目的运行时长往往很长，如果只停留在跑通的层面，那是不够的。\n",
    "\n",
    "本质是因为GPU在计算的时候，相较于cpu多了3个时间上的开销\n",
    "1. 数据传输开销 (CPU 内存 <-> GPU 显存)\n",
    "2. 核心启动开销 (GPU 核心启动时间)\n",
    "3. 性能浪费：计算量和数据批次\n",
    "\n",
    "下面详细介绍下"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd8140d3",
   "metadata": {},
   "source": [
    "### 数据传输开销 (CPU 内存 <-> GPU 显存)\n",
    "\n",
    "- 在 GPU 进行任何计算之前，数据（输入张量 X_train、y_train，模型参数）需要从计算机的主内存 (RAM) 复制到 GPU 专用的显存 (VRAM) 中。\n",
    "- 当结果传回 CPU 时（例如，使用 loss.item() 获取损失值用于打印或记录，或者获取最终预测结果），数据也需要从 GPU 显存复制回 CPU 内存。\n",
    "- 对于少量数据和非常快速的计算任务，这个传输时间可能比 GPU 通过并行计算节省下来的时间还要长。\n",
    "\n",
    "在上述代码中，循环里的 loss.item() 操作会在每个 epoch 都进行一次从 GPU 到 CPU 的数据同步和传输，以便获取标量损失值。对于20000个epoch来说，这会累积不少的传输开销。\n",
    "\n",
    "### 核心启动开销 (GPU 核心启动时间)\n",
    "- GPU 执行的每个操作（例如，一个线性层的前向传播、一个激活函数）都涉及到在 GPU 上启动一个“核心”(kernel)——一个在 GPU 众多计算单元上运行的小程序。\n",
    "- 启动每个核心都有一个小的、固定的开销。\n",
    "- 如果核心内的实际计算量非常小（本项目的小型网络和鸢尾花数据），这个启动开销在总时间中的占比就会比较大。相比之下，CPU 执行这些小操作的“调度”开销通常更低。\n",
    "\n",
    "### 性能浪费：计算量和数据批次\n",
    "- 这个数据量太少，gpu的很多计算单元都没有被用到，即使用了全批次也没有用到的全部计算单元。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ec1409c",
   "metadata": {},
   "source": [
    "\n",
    "综上，数据传输和各种固定开销的总和，超过了 GPU 在这点计算量上通过并行处理所能节省的时间，导致了 GPU 比 CPU 慢的现象。\n",
    "- CPU (12th Gen Intel Core i9-12900KF): 对于这种小任务，CPU 的单核性能强劲，且没有显著的数据传输到“另一块芯片”的开销。它可以非常迅速地完成计算。\n",
    "- GPU (NVIDIA GeForce RTX 3080 Ti):需要花费时间将数据和模型从 CPU 内存移动到 GPU 显存。\n",
    "- 每次在 GPU 上执行运算（如 model(X_train)、loss.backward()) 都有核心启动的固定开销。\n",
    "- loss.item() 在每个 epoch 都需要将结果从 GPU 传回 CPU，这在总共 20000 个 epoch 中会累积。\n",
    "- GPU 强大的并行计算能力在这种小任务上完全没有用武之地。\n",
    "\n",
    "这些特性导致GPU在处理鸢尾花分类这种“玩具级别”的问题时，它的优势无法体现，反而会因为上述开销显得“笨重”。\n",
    "\n",
    "那么什么时候 GPU 会发挥巨大优势？\n",
    "\n",
    "- 大型数据集： 例如，图像数据集成千上万张图片，每张图片维度很高。\n",
    "- 大型模型： 例如，深度卷积网络 (CNNs like ResNet, VGG) 或 Transformer 模型，它们有数百万甚至数十亿的参数，计算量巨大。\n",
    "- 合适的批处理大小： 能够充分利用 GPU 并行性的 batch size，不至于还有剩余的计算量没有被 GPU 处理。\n",
    "- 复杂的、可并行的运算： 大量的矩阵乘法、卷积等。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4151367c",
   "metadata": {},
   "source": [
    "针对上面反应的3个问题，能够优化的只有数据传输时间，针对性解决即可，很容易想到2个思路：\n",
    "1. 直接不打印训练过程的loss了，但是这样会没办法记录最后的可视化图片，只能肉眼观察loss数值变化。\n",
    "2. 每隔200个epoch保存一下loss，不需要20000个epoch每次都打印，\n",
    "\n",
    "下面先尝试第一个思路："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "27cf2fbc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [100/20000], Loss: 1.0545\n",
      "Epoch [200/20000], Loss: 1.0159\n",
      "Epoch [300/20000], Loss: 0.9747\n",
      "Epoch [400/20000], Loss: 0.9242\n",
      "Epoch [500/20000], Loss: 0.8653\n",
      "Epoch [600/20000], Loss: 0.8016\n",
      "Epoch [700/20000], Loss: 0.7381\n",
      "Epoch [800/20000], Loss: 0.6791\n",
      "Epoch [900/20000], Loss: 0.6273\n",
      "Epoch [1000/20000], Loss: 0.5832\n",
      "Epoch [1100/20000], Loss: 0.5461\n",
      "Epoch [1200/20000], Loss: 0.5148\n",
      "Epoch [1300/20000], Loss: 0.4880\n",
      "Epoch [1400/20000], Loss: 0.4648\n",
      "Epoch [1500/20000], Loss: 0.4443\n",
      "Epoch [1600/20000], Loss: 0.4258\n",
      "Epoch [1700/20000], Loss: 0.4090\n",
      "Epoch [1800/20000], Loss: 0.3934\n",
      "Epoch [1900/20000], Loss: 0.3788\n",
      "Epoch [2000/20000], Loss: 0.3652\n",
      "Epoch [2100/20000], Loss: 0.3522\n",
      "Epoch [2200/20000], Loss: 0.3399\n",
      "Epoch [2300/20000], Loss: 0.3282\n",
      "Epoch [2400/20000], Loss: 0.3170\n",
      "Epoch [2500/20000], Loss: 0.3063\n",
      "Epoch [2600/20000], Loss: 0.2960\n",
      "Epoch [2700/20000], Loss: 0.2863\n",
      "Epoch [2800/20000], Loss: 0.2770\n",
      "Epoch [2900/20000], Loss: 0.2681\n",
      "Epoch [3000/20000], Loss: 0.2596\n",
      "Epoch [3100/20000], Loss: 0.2515\n",
      "Epoch [3200/20000], Loss: 0.2438\n",
      "Epoch [3300/20000], Loss: 0.2365\n",
      "Epoch [3400/20000], Loss: 0.2296\n",
      "Epoch [3500/20000], Loss: 0.2230\n",
      "Epoch [3600/20000], Loss: 0.2167\n",
      "Epoch [3700/20000], Loss: 0.2107\n",
      "Epoch [3800/20000], Loss: 0.2050\n",
      "Epoch [3900/20000], Loss: 0.1996\n",
      "Epoch [4000/20000], Loss: 0.1944\n",
      "Epoch [4100/20000], Loss: 0.1895\n",
      "Epoch [4200/20000], Loss: 0.1849\n",
      "Epoch [4300/20000], Loss: 0.1804\n",
      "Epoch [4400/20000], Loss: 0.1762\n",
      "Epoch [4500/20000], Loss: 0.1722\n",
      "Epoch [4600/20000], Loss: 0.1684\n",
      "Epoch [4700/20000], Loss: 0.1647\n",
      "Epoch [4800/20000], Loss: 0.1612\n",
      "Epoch [4900/20000], Loss: 0.1579\n",
      "Epoch [5000/20000], Loss: 0.1547\n",
      "Epoch [5100/20000], Loss: 0.1517\n",
      "Epoch [5200/20000], Loss: 0.1487\n",
      "Epoch [5300/20000], Loss: 0.1460\n",
      "Epoch [5400/20000], Loss: 0.1433\n",
      "Epoch [5500/20000], Loss: 0.1407\n",
      "Epoch [5600/20000], Loss: 0.1383\n",
      "Epoch [5700/20000], Loss: 0.1359\n",
      "Epoch [5800/20000], Loss: 0.1337\n",
      "Epoch [5900/20000], Loss: 0.1315\n",
      "Epoch [6000/20000], Loss: 0.1295\n",
      "Epoch [6100/20000], Loss: 0.1275\n",
      "Epoch [6200/20000], Loss: 0.1255\n",
      "Epoch [6300/20000], Loss: 0.1237\n",
      "Epoch [6400/20000], Loss: 0.1219\n",
      "Epoch [6500/20000], Loss: 0.1202\n",
      "Epoch [6600/20000], Loss: 0.1186\n",
      "Epoch [6700/20000], Loss: 0.1170\n",
      "Epoch [6800/20000], Loss: 0.1155\n",
      "Epoch [6900/20000], Loss: 0.1140\n",
      "Epoch [7000/20000], Loss: 0.1126\n",
      "Epoch [7100/20000], Loss: 0.1112\n",
      "Epoch [7200/20000], Loss: 0.1099\n",
      "Epoch [7300/20000], Loss: 0.1086\n",
      "Epoch [7400/20000], Loss: 0.1074\n",
      "Epoch [7500/20000], Loss: 0.1062\n",
      "Epoch [7600/20000], Loss: 0.1050\n",
      "Epoch [7700/20000], Loss: 0.1039\n",
      "Epoch [7800/20000], Loss: 0.1028\n",
      "Epoch [7900/20000], Loss: 0.1018\n",
      "Epoch [8000/20000], Loss: 0.1007\n",
      "Epoch [8100/20000], Loss: 0.0998\n",
      "Epoch [8200/20000], Loss: 0.0988\n",
      "Epoch [8300/20000], Loss: 0.0979\n",
      "Epoch [8400/20000], Loss: 0.0970\n",
      "Epoch [8500/20000], Loss: 0.0961\n",
      "Epoch [8600/20000], Loss: 0.0953\n",
      "Epoch [8700/20000], Loss: 0.0944\n",
      "Epoch [8800/20000], Loss: 0.0936\n",
      "Epoch [8900/20000], Loss: 0.0929\n",
      "Epoch [9000/20000], Loss: 0.0921\n",
      "Epoch [9100/20000], Loss: 0.0914\n",
      "Epoch [9200/20000], Loss: 0.0907\n",
      "Epoch [9300/20000], Loss: 0.0900\n",
      "Epoch [9400/20000], Loss: 0.0893\n",
      "Epoch [9500/20000], Loss: 0.0886\n",
      "Epoch [9600/20000], Loss: 0.0880\n",
      "Epoch [9700/20000], Loss: 0.0874\n",
      "Epoch [9800/20000], Loss: 0.0868\n",
      "Epoch [9900/20000], Loss: 0.0862\n",
      "Epoch [10000/20000], Loss: 0.0856\n",
      "Epoch [10100/20000], Loss: 0.0851\n",
      "Epoch [10200/20000], Loss: 0.0845\n",
      "Epoch [10300/20000], Loss: 0.0840\n",
      "Epoch [10400/20000], Loss: 0.0835\n",
      "Epoch [10500/20000], Loss: 0.0830\n",
      "Epoch [10600/20000], Loss: 0.0825\n",
      "Epoch [10700/20000], Loss: 0.0820\n",
      "Epoch [10800/20000], Loss: 0.0815\n",
      "Epoch [10900/20000], Loss: 0.0810\n",
      "Epoch [11000/20000], Loss: 0.0806\n",
      "Epoch [11100/20000], Loss: 0.0801\n",
      "Epoch [11200/20000], Loss: 0.0797\n",
      "Epoch [11300/20000], Loss: 0.0793\n",
      "Epoch [11400/20000], Loss: 0.0789\n",
      "Epoch [11500/20000], Loss: 0.0785\n",
      "Epoch [11600/20000], Loss: 0.0781\n",
      "Epoch [11700/20000], Loss: 0.0777\n",
      "Epoch [11800/20000], Loss: 0.0773\n",
      "Epoch [11900/20000], Loss: 0.0770\n",
      "Epoch [12000/20000], Loss: 0.0766\n",
      "Epoch [12100/20000], Loss: 0.0763\n",
      "Epoch [12200/20000], Loss: 0.0759\n",
      "Epoch [12300/20000], Loss: 0.0756\n",
      "Epoch [12400/20000], Loss: 0.0752\n",
      "Epoch [12500/20000], Loss: 0.0749\n",
      "Epoch [12600/20000], Loss: 0.0746\n",
      "Epoch [12700/20000], Loss: 0.0743\n",
      "Epoch [12800/20000], Loss: 0.0740\n",
      "Epoch [12900/20000], Loss: 0.0737\n",
      "Epoch [13000/20000], Loss: 0.0734\n",
      "Epoch [13100/20000], Loss: 0.0731\n",
      "Epoch [13200/20000], Loss: 0.0728\n",
      "Epoch [13300/20000], Loss: 0.0725\n",
      "Epoch [13400/20000], Loss: 0.0723\n",
      "Epoch [13500/20000], Loss: 0.0720\n",
      "Epoch [13600/20000], Loss: 0.0717\n",
      "Epoch [13700/20000], Loss: 0.0715\n",
      "Epoch [13800/20000], Loss: 0.0712\n",
      "Epoch [13900/20000], Loss: 0.0710\n",
      "Epoch [14000/20000], Loss: 0.0707\n",
      "Epoch [14100/20000], Loss: 0.0705\n",
      "Epoch [14200/20000], Loss: 0.0703\n",
      "Epoch [14300/20000], Loss: 0.0700\n",
      "Epoch [14400/20000], Loss: 0.0698\n",
      "Epoch [14500/20000], Loss: 0.0696\n",
      "Epoch [14600/20000], Loss: 0.0694\n",
      "Epoch [14700/20000], Loss: 0.0691\n",
      "Epoch [14800/20000], Loss: 0.0689\n",
      "Epoch [14900/20000], Loss: 0.0687\n",
      "Epoch [15000/20000], Loss: 0.0685\n",
      "Epoch [15100/20000], Loss: 0.0683\n",
      "Epoch [15200/20000], Loss: 0.0681\n",
      "Epoch [15300/20000], Loss: 0.0679\n",
      "Epoch [15400/20000], Loss: 0.0677\n",
      "Epoch [15500/20000], Loss: 0.0675\n",
      "Epoch [15600/20000], Loss: 0.0674\n",
      "Epoch [15700/20000], Loss: 0.0672\n",
      "Epoch [15800/20000], Loss: 0.0670\n",
      "Epoch [15900/20000], Loss: 0.0668\n",
      "Epoch [16000/20000], Loss: 0.0666\n",
      "Epoch [16100/20000], Loss: 0.0665\n",
      "Epoch [16200/20000], Loss: 0.0663\n",
      "Epoch [16300/20000], Loss: 0.0661\n",
      "Epoch [16400/20000], Loss: 0.0660\n",
      "Epoch [16500/20000], Loss: 0.0658\n",
      "Epoch [16600/20000], Loss: 0.0656\n",
      "Epoch [16700/20000], Loss: 0.0655\n",
      "Epoch [16800/20000], Loss: 0.0653\n",
      "Epoch [16900/20000], Loss: 0.0652\n",
      "Epoch [17000/20000], Loss: 0.0650\n",
      "Epoch [17100/20000], Loss: 0.0649\n",
      "Epoch [17200/20000], Loss: 0.0647\n",
      "Epoch [17300/20000], Loss: 0.0646\n",
      "Epoch [17400/20000], Loss: 0.0644\n",
      "Epoch [17500/20000], Loss: 0.0643\n",
      "Epoch [17600/20000], Loss: 0.0642\n",
      "Epoch [17700/20000], Loss: 0.0640\n",
      "Epoch [17800/20000], Loss: 0.0639\n",
      "Epoch [17900/20000], Loss: 0.0638\n",
      "Epoch [18000/20000], Loss: 0.0636\n",
      "Epoch [18100/20000], Loss: 0.0635\n",
      "Epoch [18200/20000], Loss: 0.0634\n",
      "Epoch [18300/20000], Loss: 0.0633\n",
      "Epoch [18400/20000], Loss: 0.0631\n",
      "Epoch [18500/20000], Loss: 0.0630\n",
      "Epoch [18600/20000], Loss: 0.0629\n",
      "Epoch [18700/20000], Loss: 0.0628\n",
      "Epoch [18800/20000], Loss: 0.0626\n",
      "Epoch [18900/20000], Loss: 0.0625\n",
      "Epoch [19000/20000], Loss: 0.0624\n",
      "Epoch [19100/20000], Loss: 0.0623\n",
      "Epoch [19200/20000], Loss: 0.0622\n",
      "Epoch [19300/20000], Loss: 0.0621\n",
      "Epoch [19400/20000], Loss: 0.0620\n",
      "Epoch [19500/20000], Loss: 0.0619\n",
      "Epoch [19600/20000], Loss: 0.0618\n",
      "Epoch [19700/20000], Loss: 0.0616\n",
      "Epoch [19800/20000], Loss: 0.0615\n",
      "Epoch [19900/20000], Loss: 0.0614\n",
      "Epoch [20000/20000], Loss: 0.0613\n",
      "Training time: 2.86 seconds\n"
     ]
    }
   ],
   "source": [
    "# 知道了哪里耗时，针对性优化一下\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "\n",
    "# 仍然用4特征，3分类的鸢尾花数据集作为我们今天的数据集\n",
    "# 加载鸢尾花数据集\n",
    "iris = load_iris()\n",
    "X = iris.data  # 特征数据\n",
    "y = iris.target  # 标签数据\n",
    "# 划分训练集和测试集\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# # 打印下尺寸\n",
    "# print(X_train.shape)\n",
    "# print(y_train.shape)\n",
    "# print(X_test.shape)\n",
    "# print(y_test.shape)\n",
    "\n",
    "# 归一化数据，神经网络对于输入数据的尺寸敏感，归一化是最常见的处理方式\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "scaler = MinMaxScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test) #确保训练集和测试集是相同的缩放\n",
    "\n",
    "\n",
    "# 将数据转换为 PyTorch 张量，因为 PyTorch 使用张量进行训练\n",
    "# y_train和y_test是整数，所以需要转化为long类型，如果是float32，会输出1.0 0.0\n",
    "X_train = torch.FloatTensor(X_train)\n",
    "y_train = torch.LongTensor(y_train)\n",
    "X_test = torch.FloatTensor(X_test)\n",
    "y_test = torch.LongTensor(y_test)\n",
    "\n",
    "class MLP(nn.Module): # 定义一个多层感知机（MLP）模型，继承父类nn.Module\n",
    "    def __init__(self): # 初始化函数\n",
    "        super(MLP, self).__init__() # 调用父类的初始化函数\n",
    " # 前三行是八股文，后面的是自定义的\n",
    "\n",
    "        self.fc1 = nn.Linear(4, 10)  # 输入层到隐藏层\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(10, 3)  # 隐藏层到输出层\n",
    "# 输出层不需要激活函数，因为后面会用到交叉熵函数cross_entropy，交叉熵函数内部有softmax函数，会把输出转化为概率\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.fc1(x)\n",
    "        out = self.relu(out)\n",
    "        out = self.fc2(out)\n",
    "        return out\n",
    "\n",
    "# 实例化模型\n",
    "model = MLP()\n",
    "\n",
    "# 分类问题使用交叉熵损失函数\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# 使用随机梯度下降优化器\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)\n",
    "\n",
    "# # 使用自适应学习率的化器\n",
    "# optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "\n",
    "# 训练模型\n",
    "num_epochs = 20000 # 训练的轮数\n",
    "\n",
    "# 用于存储每个 epoch 的损失值\n",
    "losses = []\n",
    "\n",
    "import time\n",
    "start_time = time.time() # 记录开始时间\n",
    "\n",
    "for epoch in range(num_epochs): # range是从0开始，所以epoch是从0开始\n",
    "    # 前向传播\n",
    "    outputs = model.forward(X_train)   # 显式调用forward函数\n",
    "    # outputs = model(X_train)  # 常见写法隐式调用forward函数，其实是用了model类的__call__方法\n",
    "    loss = criterion(outputs, y_train) # output是模型预测值，y_train是真实标签\n",
    "\n",
    "    # 反向传播和优化\n",
    "    optimizer.zero_grad() #梯度清零，因为PyTorch会累积梯度，所以每次迭代需要清零，梯度累计是那种小的bitchsize模拟大的bitchsize\n",
    "    loss.backward() # 反向传播计算梯度\n",
    "    optimizer.step() # 更新参数\n",
    "\n",
    "    # 记录损失值\n",
    "    # losses.append(loss.item())\n",
    "\n",
    "    # 打印训练信息\n",
    "    if (epoch + 1) % 100 == 0: # range是从0开始，所以epoch+1是从当前epoch开始，每100个epoch打印一次\n",
    "        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')\n",
    "\n",
    "time_all = time.time() - start_time # 计算训练时间\n",
    "print(f'Training time: {time_all:.2f} seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e88d65c",
   "metadata": {},
   "source": [
    "优化后发现确实效果好，近乎和用cpu训练的时长差不多。所以可以理解为数据从gpu到cpu的传输占用了大量时间。\n",
    "\n",
    "下面尝试下第二个思路："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da2727cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "使用设备: cuda:0\n",
      "Epoch [100/20000], Loss: 1.0107\n",
      "Epoch [200/20000], Loss: 0.9498\n",
      "Epoch [200/20000], Loss: 0.9498\n",
      "Epoch [300/20000], Loss: 0.8876\n",
      "Epoch [400/20000], Loss: 0.8256\n",
      "Epoch [400/20000], Loss: 0.8256\n",
      "Epoch [500/20000], Loss: 0.7634\n",
      "Epoch [600/20000], Loss: 0.7044\n",
      "Epoch [600/20000], Loss: 0.7044\n",
      "Epoch [700/20000], Loss: 0.6521\n",
      "Epoch [800/20000], Loss: 0.6073\n",
      "Epoch [800/20000], Loss: 0.6073\n",
      "Epoch [900/20000], Loss: 0.5698\n",
      "Epoch [1000/20000], Loss: 0.5385\n",
      "Epoch [1000/20000], Loss: 0.5385\n",
      "Epoch [1100/20000], Loss: 0.5121\n",
      "Epoch [1200/20000], Loss: 0.4896\n",
      "Epoch [1200/20000], Loss: 0.4896\n",
      "Epoch [1300/20000], Loss: 0.4701\n",
      "Epoch [1400/20000], Loss: 0.4528\n",
      "Epoch [1400/20000], Loss: 0.4528\n",
      "Epoch [1500/20000], Loss: 0.4373\n",
      "Epoch [1600/20000], Loss: 0.4231\n",
      "Epoch [1600/20000], Loss: 0.4231\n",
      "Epoch [1700/20000], Loss: 0.4100\n",
      "Epoch [1800/20000], Loss: 0.3977\n",
      "Epoch [1800/20000], Loss: 0.3977\n",
      "Epoch [1900/20000], Loss: 0.3861\n",
      "Epoch [2000/20000], Loss: 0.3750\n",
      "Epoch [2000/20000], Loss: 0.3750\n",
      "Epoch [2100/20000], Loss: 0.3643\n",
      "Epoch [2200/20000], Loss: 0.3540\n",
      "Epoch [2200/20000], Loss: 0.3540\n",
      "Epoch [2300/20000], Loss: 0.3441\n",
      "Epoch [2400/20000], Loss: 0.3344\n",
      "Epoch [2400/20000], Loss: 0.3344\n",
      "Epoch [2500/20000], Loss: 0.3250\n",
      "Epoch [2600/20000], Loss: 0.3158\n",
      "Epoch [2600/20000], Loss: 0.3158\n",
      "Epoch [2700/20000], Loss: 0.3069\n",
      "Epoch [2800/20000], Loss: 0.2983\n",
      "Epoch [2800/20000], Loss: 0.2983\n",
      "Epoch [2900/20000], Loss: 0.2900\n",
      "Epoch [3000/20000], Loss: 0.2818\n",
      "Epoch [3000/20000], Loss: 0.2818\n",
      "Epoch [3100/20000], Loss: 0.2739\n",
      "Epoch [3200/20000], Loss: 0.2664\n",
      "Epoch [3200/20000], Loss: 0.2664\n",
      "Epoch [3300/20000], Loss: 0.2590\n",
      "Epoch [3400/20000], Loss: 0.2519\n",
      "Epoch [3400/20000], Loss: 0.2519\n",
      "Epoch [3500/20000], Loss: 0.2452\n",
      "Epoch [3600/20000], Loss: 0.2386\n",
      "Epoch [3600/20000], Loss: 0.2386\n",
      "Epoch [3700/20000], Loss: 0.2323\n",
      "Epoch [3800/20000], Loss: 0.2262\n",
      "Epoch [3800/20000], Loss: 0.2262\n",
      "Epoch [3900/20000], Loss: 0.2204\n",
      "Epoch [4000/20000], Loss: 0.2148\n",
      "Epoch [4000/20000], Loss: 0.2148\n",
      "Epoch [4100/20000], Loss: 0.2095\n",
      "Epoch [4200/20000], Loss: 0.2044\n",
      "Epoch [4200/20000], Loss: 0.2044\n",
      "Epoch [4300/20000], Loss: 0.1995\n",
      "Epoch [4400/20000], Loss: 0.1948\n",
      "Epoch [4400/20000], Loss: 0.1948\n",
      "Epoch [4500/20000], Loss: 0.1903\n",
      "Epoch [4600/20000], Loss: 0.1859\n",
      "Epoch [4600/20000], Loss: 0.1859\n",
      "Epoch [4700/20000], Loss: 0.1818\n",
      "Epoch [4800/20000], Loss: 0.1779\n",
      "Epoch [4800/20000], Loss: 0.1779\n",
      "Epoch [4900/20000], Loss: 0.1740\n",
      "Epoch [5000/20000], Loss: 0.1704\n",
      "Epoch [5000/20000], Loss: 0.1704\n",
      "Epoch [5100/20000], Loss: 0.1670\n",
      "Epoch [5200/20000], Loss: 0.1636\n",
      "Epoch [5200/20000], Loss: 0.1636\n",
      "Epoch [5300/20000], Loss: 0.1604\n",
      "Epoch [5400/20000], Loss: 0.1574\n",
      "Epoch [5400/20000], Loss: 0.1574\n",
      "Epoch [5500/20000], Loss: 0.1544\n",
      "Epoch [5600/20000], Loss: 0.1516\n",
      "Epoch [5600/20000], Loss: 0.1516\n",
      "Epoch [5700/20000], Loss: 0.1490\n",
      "Epoch [5800/20000], Loss: 0.1463\n",
      "Epoch [5800/20000], Loss: 0.1463\n",
      "Epoch [5900/20000], Loss: 0.1438\n",
      "Epoch [6000/20000], Loss: 0.1415\n",
      "Epoch [6000/20000], Loss: 0.1415\n",
      "Epoch [6100/20000], Loss: 0.1391\n",
      "Epoch [6200/20000], Loss: 0.1369\n",
      "Epoch [6200/20000], Loss: 0.1369\n",
      "Epoch [6300/20000], Loss: 0.1348\n",
      "Epoch [6400/20000], Loss: 0.1327\n",
      "Epoch [6400/20000], Loss: 0.1327\n",
      "Epoch [6500/20000], Loss: 0.1307\n",
      "Epoch [6600/20000], Loss: 0.1289\n",
      "Epoch [6600/20000], Loss: 0.1289\n",
      "Epoch [6700/20000], Loss: 0.1270\n",
      "Epoch [6800/20000], Loss: 0.1252\n",
      "Epoch [6800/20000], Loss: 0.1252\n",
      "Epoch [6900/20000], Loss: 0.1235\n",
      "Epoch [7000/20000], Loss: 0.1219\n",
      "Epoch [7000/20000], Loss: 0.1219\n",
      "Epoch [7100/20000], Loss: 0.1203\n",
      "Epoch [7200/20000], Loss: 0.1187\n",
      "Epoch [7200/20000], Loss: 0.1187\n",
      "Epoch [7300/20000], Loss: 0.1173\n",
      "Epoch [7400/20000], Loss: 0.1159\n",
      "Epoch [7400/20000], Loss: 0.1159\n",
      "Epoch [7500/20000], Loss: 0.1145\n",
      "Epoch [7600/20000], Loss: 0.1131\n",
      "Epoch [7600/20000], Loss: 0.1131\n",
      "Epoch [7700/20000], Loss: 0.1119\n",
      "Epoch [7800/20000], Loss: 0.1106\n",
      "Epoch [7800/20000], Loss: 0.1106\n",
      "Epoch [7900/20000], Loss: 0.1093\n",
      "Epoch [8000/20000], Loss: 0.1082\n",
      "Epoch [8000/20000], Loss: 0.1082\n",
      "Epoch [8100/20000], Loss: 0.1071\n",
      "Epoch [8200/20000], Loss: 0.1059\n",
      "Epoch [8200/20000], Loss: 0.1059\n",
      "Epoch [8300/20000], Loss: 0.1049\n",
      "Epoch [8400/20000], Loss: 0.1038\n",
      "Epoch [8400/20000], Loss: 0.1038\n",
      "Epoch [8500/20000], Loss: 0.1028\n",
      "Epoch [8600/20000], Loss: 0.1019\n",
      "Epoch [8600/20000], Loss: 0.1019\n",
      "Epoch [8700/20000], Loss: 0.1009\n",
      "Epoch [8800/20000], Loss: 0.1000\n",
      "Epoch [8800/20000], Loss: 0.1000\n",
      "Epoch [8900/20000], Loss: 0.0991\n",
      "Epoch [9000/20000], Loss: 0.0982\n",
      "Epoch [9000/20000], Loss: 0.0982\n",
      "Epoch [9100/20000], Loss: 0.0974\n",
      "Epoch [9200/20000], Loss: 0.0966\n",
      "Epoch [9200/20000], Loss: 0.0966\n",
      "Epoch [9300/20000], Loss: 0.0958\n",
      "Epoch [9400/20000], Loss: 0.0950\n",
      "Epoch [9400/20000], Loss: 0.0950\n",
      "Epoch [9500/20000], Loss: 0.0943\n",
      "Epoch [9600/20000], Loss: 0.0935\n",
      "Epoch [9600/20000], Loss: 0.0935\n",
      "Epoch [9700/20000], Loss: 0.0928\n",
      "Epoch [9800/20000], Loss: 0.0921\n",
      "Epoch [9800/20000], Loss: 0.0921\n",
      "Epoch [9900/20000], Loss: 0.0914\n",
      "Epoch [10000/20000], Loss: 0.0907\n",
      "Epoch [10000/20000], Loss: 0.0907\n",
      "Epoch [10100/20000], Loss: 0.0901\n",
      "Epoch [10200/20000], Loss: 0.0895\n",
      "Epoch [10200/20000], Loss: 0.0895\n",
      "Epoch [10300/20000], Loss: 0.0888\n",
      "Epoch [10400/20000], Loss: 0.0883\n",
      "Epoch [10400/20000], Loss: 0.0883\n",
      "Epoch [10500/20000], Loss: 0.0877\n",
      "Epoch [10600/20000], Loss: 0.0871\n",
      "Epoch [10600/20000], Loss: 0.0871\n",
      "Epoch [10700/20000], Loss: 0.0865\n",
      "Epoch [10800/20000], Loss: 0.0860\n",
      "Epoch [10800/20000], Loss: 0.0860\n",
      "Epoch [10900/20000], Loss: 0.0855\n",
      "Epoch [11000/20000], Loss: 0.0850\n",
      "Epoch [11000/20000], Loss: 0.0850\n",
      "Epoch [11100/20000], Loss: 0.0845\n",
      "Epoch [11200/20000], Loss: 0.0840\n",
      "Epoch [11200/20000], Loss: 0.0840\n",
      "Epoch [11300/20000], Loss: 0.0835\n",
      "Epoch [11400/20000], Loss: 0.0830\n",
      "Epoch [11400/20000], Loss: 0.0830\n",
      "Epoch [11500/20000], Loss: 0.0826\n",
      "Epoch [11600/20000], Loss: 0.0822\n",
      "Epoch [11600/20000], Loss: 0.0822\n",
      "Epoch [11700/20000], Loss: 0.0817\n",
      "Epoch [11800/20000], Loss: 0.0813\n",
      "Epoch [11800/20000], Loss: 0.0813\n",
      "Epoch [11900/20000], Loss: 0.0808\n",
      "Epoch [12000/20000], Loss: 0.0805\n",
      "Epoch [12000/20000], Loss: 0.0805\n",
      "Epoch [12100/20000], Loss: 0.0801\n",
      "Epoch [12200/20000], Loss: 0.0797\n",
      "Epoch [12200/20000], Loss: 0.0797\n",
      "Epoch [12300/20000], Loss: 0.0793\n",
      "Epoch [12400/20000], Loss: 0.0789\n",
      "Epoch [12400/20000], Loss: 0.0789\n",
      "Epoch [12500/20000], Loss: 0.0785\n",
      "Epoch [12600/20000], Loss: 0.0781\n",
      "Epoch [12600/20000], Loss: 0.0781\n",
      "Epoch [12700/20000], Loss: 0.0778\n",
      "Epoch [12800/20000], Loss: 0.0775\n",
      "Epoch [12800/20000], Loss: 0.0775\n",
      "Epoch [12900/20000], Loss: 0.0771\n",
      "Epoch [13000/20000], Loss: 0.0768\n",
      "Epoch [13000/20000], Loss: 0.0768\n",
      "Epoch [13100/20000], Loss: 0.0765\n",
      "Epoch [13200/20000], Loss: 0.0761\n",
      "Epoch [13200/20000], Loss: 0.0761\n",
      "Epoch [13300/20000], Loss: 0.0758\n",
      "Epoch [13400/20000], Loss: 0.0755\n",
      "Epoch [13400/20000], Loss: 0.0755\n",
      "Epoch [13500/20000], Loss: 0.0752\n",
      "Epoch [13600/20000], Loss: 0.0749\n",
      "Epoch [13600/20000], Loss: 0.0749\n",
      "Epoch [13700/20000], Loss: 0.0746\n",
      "Epoch [13800/20000], Loss: 0.0743\n",
      "Epoch [13800/20000], Loss: 0.0743\n",
      "Epoch [13900/20000], Loss: 0.0740\n",
      "Epoch [14000/20000], Loss: 0.0737\n",
      "Epoch [14000/20000], Loss: 0.0737\n",
      "Epoch [14100/20000], Loss: 0.0735\n",
      "Epoch [14200/20000], Loss: 0.0732\n",
      "Epoch [14200/20000], Loss: 0.0732\n",
      "Epoch [14300/20000], Loss: 0.0729\n",
      "Epoch [14400/20000], Loss: 0.0727\n",
      "Epoch [14400/20000], Loss: 0.0727\n",
      "Epoch [14500/20000], Loss: 0.0724\n",
      "Epoch [14600/20000], Loss: 0.0722\n",
      "Epoch [14600/20000], Loss: 0.0722\n",
      "Epoch [14700/20000], Loss: 0.0720\n",
      "Epoch [14800/20000], Loss: 0.0717\n",
      "Epoch [14800/20000], Loss: 0.0717\n",
      "Epoch [14900/20000], Loss: 0.0715\n",
      "Epoch [15000/20000], Loss: 0.0712\n",
      "Epoch [15000/20000], Loss: 0.0712\n",
      "Epoch [15100/20000], Loss: 0.0710\n",
      "Epoch [15200/20000], Loss: 0.0708\n",
      "Epoch [15200/20000], Loss: 0.0708\n",
      "Epoch [15300/20000], Loss: 0.0706\n",
      "Epoch [15400/20000], Loss: 0.0704\n",
      "Epoch [15400/20000], Loss: 0.0704\n",
      "Epoch [15500/20000], Loss: 0.0701\n",
      "Epoch [15600/20000], Loss: 0.0699\n",
      "Epoch [15600/20000], Loss: 0.0699\n",
      "Epoch [15700/20000], Loss: 0.0697\n",
      "Epoch [15800/20000], Loss: 0.0695\n",
      "Epoch [15800/20000], Loss: 0.0695\n",
      "Epoch [15900/20000], Loss: 0.0693\n",
      "Epoch [16000/20000], Loss: 0.0691\n",
      "Epoch [16000/20000], Loss: 0.0691\n",
      "Epoch [16100/20000], Loss: 0.0689\n",
      "Epoch [16200/20000], Loss: 0.0687\n",
      "Epoch [16200/20000], Loss: 0.0687\n",
      "Epoch [16300/20000], Loss: 0.0685\n",
      "Epoch [16400/20000], Loss: 0.0684\n",
      "Epoch [16400/20000], Loss: 0.0684\n",
      "Epoch [16500/20000], Loss: 0.0682\n",
      "Epoch [16600/20000], Loss: 0.0680\n",
      "Epoch [16600/20000], Loss: 0.0680\n",
      "Epoch [16700/20000], Loss: 0.0678\n",
      "Epoch [16800/20000], Loss: 0.0676\n",
      "Epoch [16800/20000], Loss: 0.0676\n",
      "Epoch [16900/20000], Loss: 0.0675\n",
      "Epoch [17000/20000], Loss: 0.0673\n",
      "Epoch [17000/20000], Loss: 0.0673\n",
      "Epoch [17100/20000], Loss: 0.0671\n",
      "Epoch [17200/20000], Loss: 0.0669\n",
      "Epoch [17200/20000], Loss: 0.0669\n",
      "Epoch [17300/20000], Loss: 0.0668\n",
      "Epoch [17400/20000], Loss: 0.0667\n",
      "Epoch [17400/20000], Loss: 0.0667\n",
      "Epoch [17500/20000], Loss: 0.0664\n",
      "Epoch [17600/20000], Loss: 0.0663\n",
      "Epoch [17600/20000], Loss: 0.0663\n",
      "Epoch [17700/20000], Loss: 0.0662\n",
      "Epoch [17800/20000], Loss: 0.0660\n",
      "Epoch [17800/20000], Loss: 0.0660\n",
      "Epoch [17900/20000], Loss: 0.0658\n",
      "Epoch [18000/20000], Loss: 0.0657\n",
      "Epoch [18000/20000], Loss: 0.0657\n",
      "Epoch [18100/20000], Loss: 0.0655\n",
      "Epoch [18200/20000], Loss: 0.0654\n",
      "Epoch [18200/20000], Loss: 0.0654\n",
      "Epoch [18300/20000], Loss: 0.0653\n",
      "Epoch [18400/20000], Loss: 0.0651\n",
      "Epoch [18400/20000], Loss: 0.0651\n",
      "Epoch [18500/20000], Loss: 0.0650\n",
      "Epoch [18600/20000], Loss: 0.0649\n",
      "Epoch [18600/20000], Loss: 0.0649\n",
      "Epoch [18700/20000], Loss: 0.0647\n",
      "Epoch [18800/20000], Loss: 0.0646\n",
      "Epoch [18800/20000], Loss: 0.0646\n",
      "Epoch [18900/20000], Loss: 0.0645\n",
      "Epoch [19000/20000], Loss: 0.0644\n",
      "Epoch [19000/20000], Loss: 0.0644\n",
      "Epoch [19100/20000], Loss: 0.0643\n",
      "Epoch [19200/20000], Loss: 0.0641\n",
      "Epoch [19200/20000], Loss: 0.0641\n",
      "Epoch [19300/20000], Loss: 0.0640\n",
      "Epoch [19400/20000], Loss: 0.0638\n",
      "Epoch [19400/20000], Loss: 0.0638\n",
      "Epoch [19500/20000], Loss: 0.0637\n",
      "Epoch [19600/20000], Loss: 0.0636\n",
      "Epoch [19600/20000], Loss: 0.0636\n",
      "Epoch [19700/20000], Loss: 0.0635\n",
      "Epoch [19800/20000], Loss: 0.0634\n",
      "Epoch [19800/20000], Loss: 0.0634\n",
      "Epoch [19900/20000], Loss: 0.0632\n",
      "Epoch [20000/20000], Loss: 0.0631\n",
      "Epoch [20000/20000], Loss: 0.0631\n",
      "Training time: 10.38 seconds\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/xnp5ZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABJNUlEQVR4nO3deXxU1f3/8fcsmcmeELKxBAJBAUEWWSKC4pIKaHFDRUplsRVFtCq1rZYqLl/FvdaqUKl7bUX9KW6AAkItimwCArKIrAJJCCF7MpPM3N8fSQZHtpBM5iaT1/PxuI/MnDl35pMbIG/OPfdci2EYhgAAAEKE1ewCAAAAAolwAwAAQgrhBgAAhBTCDQAACCmEGwAAEFIINwAAIKQQbgAAQEgh3AAAgJBCuAEAACGFcAM0IRMmTFB6enq99r3//vtlsVgCWxBwDK+++qosFotWr15tdinAMRFugDqwWCx12pYuXWp2qaaYMGGCoqOjzS4jZNSGh+NtX3/9tdklAk2a3ewCgObgjTfe8Hv++uuva+HChUe1d+/evUGfM3v2bHm93nrt+5e//EV33313gz4fTcuDDz6oTp06HdXepUsXE6oBmg/CDVAHv/71r/2ef/3111q4cOFR7T9XVlamyMjIOn9OWFhYveqTJLvdLrudv9LNRWlpqaKiok7YZ8SIEerfv3+QKgJCB6elgAA5//zz1bNnT61Zs0bnnXeeIiMj9ec//1mS9MEHH+jSSy9V27Zt5XQ6lZGRoYceekgej8fvPX4+52bXrl2yWCx68skn9eKLLyojI0NOp1MDBgzQqlWr/PY91pwbi8WiW2+9VXPnzlXPnj3ldDrVo0cPLViw4Kj6ly5dqv79+ys8PFwZGRn6xz/+EfB5PO+884769euniIgIJSYm6te//rX27dvn1yc7O1sTJ05U+/bt5XQ61aZNG11++eXatWuXr8/q1as1bNgwJSYmKiIiQp06ddINN9xQpxpeeOEF9ejRQ06nU23bttWUKVNUUFDge/3WW29VdHS0ysrKjtp3zJgxSk1N9fu5zZ8/X+eee66ioqIUExOjSy+9VJs2bfLbr/a03Q8//KBLLrlEMTExGjt2bJ3qPZGf/vn461//qo4dOyoiIkJDhw7Vxo0bj+r/+eef+2qNj4/X5Zdfrs2bNx/Vb9++ffrNb37j+/PaqVMnTZ48WW6326+fy+XS1KlTlZSUpKioKF155ZU6ePCgX5+G/KyA+uK/eUAAHTp0SCNGjNB1112nX//610pJSZFUPYciOjpaU6dOVXR0tD7//HPdd999Kioq0hNPPHHS9/33v/+t4uJi3XTTTbJYLHr88cd11VVXaceOHScd7Vm2bJnee+893XLLLYqJidGzzz6rUaNGac+ePWrdurUkae3atRo+fLjatGmjBx54QB6PRw8++KCSkpIaflBqvPrqq5o4caIGDBigGTNmKCcnR3/729/05Zdfau3atYqPj5ckjRo1Sps2bdJtt92m9PR05ebmauHChdqzZ4/v+cUXX6ykpCTdfffdio+P165du/Tee++dtIb7779fDzzwgLKysjR58mRt3bpVM2fO1KpVq/Tll18qLCxMo0eP1vPPP69PPvlE11xzjW/fsrIyffTRR5owYYJsNpuk6tOV48eP17Bhw/TYY4+prKxMM2fO1JAhQ7R27Vq/oFpVVaVhw4ZpyJAhevLJJ+s0oldYWKi8vDy/NovF4vu51Xr99ddVXFysKVOmqKKiQn/729904YUXasOGDb4/g4sWLdKIESPUuXNn3X///SovL9ff//53DR48WN98842v1v3792vgwIEqKCjQpEmT1K1bN+3bt0/vvvuuysrK5HA4fJ972223qVWrVpo+fbp27dqlZ555RrfeeqvmzJkjSQ36WQENYgA4ZVOmTDF+/tdn6NChhiRj1qxZR/UvKys7qu2mm24yIiMjjYqKCl/b+PHjjY4dO/qe79y505BktG7d2sjPz/e1f/DBB4Yk46OPPvK1TZ8+/aiaJBkOh8PYvn27r239+vWGJOPvf/+7r23kyJFGZGSksW/fPl/b999/b9jt9qPe81jGjx9vREVFHfd1t9ttJCcnGz179jTKy8t97R9//LEhybjvvvsMwzCMw4cPG5KMJ5544rjv9f777xuSjFWrVp20rp/Kzc01HA6HcfHFFxsej8fX/txzzxmSjJdfftkwDMPwer1Gu3btjFGjRvnt//bbbxuSjC+++MIwDMMoLi424uPjjRtvvNGvX3Z2thEXF+fXPn78eEOScffdd9ep1ldeecWQdMzN6XT6+tX++YiIiDB+/PFHX/uKFSsMScadd97pa+vTp4+RnJxsHDp0yNe2fv16w2q1GuPGjfO1jRs3zrBarcc8vl6v16++rKwsX5thGMadd95p2Gw2o6CgwDCM+v+sgIbitBQQQE6nUxMnTjyqPSIiwve4uLhYeXl5Ovfcc1VWVqYtW7ac9H1Hjx6tVq1a+Z6fe+65kqQdO3acdN+srCxlZGT4nvfq1UuxsbG+fT0ejxYtWqQrrrhCbdu29fXr0qWLRowYcdL3r4vVq1crNzdXt9xyi8LDw33tl156qbp166ZPPvlEUvVxcjgcWrp0qQ4fPnzM96od4fn4449VWVlZ5xoWLVokt9utO+64Q1brkX/6brzxRsXGxvpqsFgsuuaaazRv3jyVlJT4+s2ZM0ft2rXTkCFDJEkLFy5UQUGBxowZo7y8PN9ms9mUmZmpJUuWHFXD5MmT61yvJD3//PNauHCh3zZ//vyj+l1xxRVq166d7/nAgQOVmZmpefPmSZIOHDigdevWacKECUpISPD169Wrl37xi1/4+nm9Xs2dO1cjR4485lyfn5+inDRpkl/bueeeK4/Ho927d0uq/88KaCjCDRBA7dq18xu2r7Vp0yZdeeWViouLU2xsrJKSknyTkQsLC0/6vh06dPB7Xht0jhcATrRv7f61++bm5qq8vPyYV+AE6qqc2l92Xbt2Peq1bt26+V53Op167LHHNH/+fKWkpOi8887T448/ruzsbF//oUOHatSoUXrggQeUmJioyy+/XK+88opcLle9anA4HOrcubPvdak6TJaXl+vDDz+UJJWUlGjevHm65pprfL/Mv//+e0nShRdeqKSkJL/ts88+U25urt/n2O12tW/f/uQH6ycGDhyorKwsv+2CCy44qt9pp512VNvpp5/um6d0ouPfvXt35eXlqbS0VAcPHlRRUZF69uxZp/pO9ueyvj8roKEIN0AA/XSEplZBQYGGDh2q9evX68EHH9RHH32khQsX6rHHHpOkOl36XTvH4+cMw2jUfc1wxx13aNu2bZoxY4bCw8N17733qnv37lq7dq2k6tGDd999V8uXL9ett96qffv26YYbblC/fv38Rloa4uyzz1Z6errefvttSdJHH32k8vJyjR492ten9uf2xhtvHDW6snDhQn3wwQd+7+l0Ov1GjELByf5sBeNnBRxLaP1NA5qgpUuX6tChQ3r11Vd1++2365e//KWysrL8TjOZKTk5WeHh4dq+fftRrx2rrT46duwoSdq6detRr23dutX3eq2MjAz9/ve/12effaaNGzfK7Xbrqaee8utz9tln6+GHH9bq1av15ptvatOmTXrrrbdOuQa3262dO3ceVcO1116rBQsWqKioSHPmzFF6errOPvtsvxql6uP389GVrKwsnX/++Sc5KoFTO4r0U9u2bfNNEj7R8d+yZYsSExMVFRWlpKQkxcbGHvNKq4Y41Z8V0FCEG6CR1f7v9qcjJW63Wy+88IJZJfmx2WzKysrS3LlztX//fl/79u3bjzm/oz769++v5ORkzZo1y++UxPz587V582ZdeumlkqqvSKqoqPDbNyMjQzExMb79Dh8+fNSoU58+fSTphKc7srKy5HA49Oyzz/rt/9JLL6mwsNBXQ63Ro0fL5XLptdde04IFC3Tttdf6vT5s2DDFxsbqkUceOeZ8kp9fEt2Y5s6d63dJ/cqVK7VixQrfnKk2bdqoT58+eu211/wue9+4caM+++wzXXLJJZIkq9WqK664Qh999NExb61wqqN99f1ZAQ3FpeBAIzvnnHPUqlUrjR8/Xr/73e9ksVj0xhtvNKnTQvfff78+++wzDR48WJMnT5bH49Fzzz2nnj17at26dXV6j8rKSv3f//3fUe0JCQm65ZZb9Nhjj2nixIkaOnSoxowZ47sUPD09XXfeeaek6tGGiy66SNdee63OOOMM2e12vf/++8rJydF1110nSXrttdf0wgsv6Morr1RGRoaKi4s1e/ZsxcbG+n5JH0tSUpLuuecePfDAAxo+fLguu+wybd26VS+88IIGDBhw1IKMZ511lrp06aJp06bJ5XL5nZKSpNjYWM2cOVPXX3+9zjrrLF133XVKSkrSnj179Mknn2jw4MF67rnn6nTsjmf+/PnHnHB+zjnnqHPnzr7nXbp00ZAhQzR58mS5XC4988wzat26tf74xz/6+jzxxBMaMWKEBg0apN/85je+S8Hj4uJ0//33+/o98sgj+uyzzzR06FBNmjRJ3bt314EDB/TOO+9o2bJlvknCdVHfnxXQYKZdpwU0Y8e7FLxHjx7H7P/ll18aZ599thEREWG0bdvW+OMf/2h8+umnhiRjyZIlvn7HuxT8WJdGSzKmT5/ue368S8GnTJly1L4dO3Y0xo8f79e2ePFio2/fvobD4TAyMjKMf/7zn8bvf/97Izw8/DhH4YjaS52PtWVkZPj6zZkzx+jbt6/hdDqNhIQEY+zYsX6XMOfl5RlTpkwxunXrZkRFRRlxcXFGZmam8fbbb/v6fPPNN8aYMWOMDh06GE6n00hOTjZ++ctfGqtXrz5pnYZRfel3t27djLCwMCMlJcWYPHmycfjw4WP2nTZtmiHJ6NKly3Hfb8mSJcawYcOMuLg4Izw83MjIyDAmTJjgV8/JLpX/uRNdCi7JeOWVVwzD8P/z8dRTTxlpaWmG0+k0zj33XGP9+vVHve+iRYuMwYMHGxEREUZsbKwxcuRI47vvvjuq3+7du41x48YZSUlJhtPpNDp37mxMmTLFcLlcfvX9/BLvJUuW+P2ZbujPCqgvi2E0of8+AmhSrrjiCm3atOmYczpgvl27dqlTp0564okndNddd5ldDtBkMOcGgCSpvLzc7/n333+vefPmBXViLAAEAnNuAEiSOnfurAkTJvjWfJk5c6YcDoffvA0AaA4INwAkScOHD9d//vMfZWdny+l0atCgQXrkkUeOuUAcADRlzLkBAAAhhTk3AAAgpBBuAABASGlxc268Xq/279+vmJiYo+5wCwAAmibDMFRcXKy2bdue9D5tLS7c7N+/X2lpaWaXAQAA6mHv3r1q3779Cfu0uHATExMjqfrgxMbGmlwNAACoi6KiIqWlpfl+j59Iiws3taeiYmNjCTcAADQzdZlSwoRiAAAQUgg3AAAgpBBuAABASCHcAACAkEK4AQAAIYVwAwAAQgrhBgAAhBTCDQAACCmEGwAAEFIINwAAIKQQbgAAQEgh3AAAgJBCuAkQj9dQblGFduWVml0KAAAtGuEmQJb/cEgDH1msSW+sNrsUAABaNMJNgCTHOiVJB4tdJlcCAEDLRrgJkKTo6nBzuKxS7iqvydUAANByEW4CJD4yTA5b9eE8WMLoDQAAZiHcBIjFYlFSTPXoTW5RhcnVAADQchFuAsgXbph3AwCAaQg3AVQbbphUDACAeQg3AZTMyA0AAKYj3ARQcky4JOlgMXNuAAAwC+EmgGrXusktYuQGAACzEG4CqHatGy4FBwDAPISbAGLkBgAA8xFuAqh2zk1eiUter2FyNQAAtEyEmwBKjHbIYpGqvIbyy9xmlwMAQItEuAkgu82q1lEOSZyaAgDALISbAEtkUjEAAKYi3ARYcmz1vBvuLwUAgDkINwHGKsUAAJiLcBNgydxfCgAAUxFuAoybZwIAYC7CTYDVrnWTy/2lAAAwBeEmwHyrFDNyAwCAKQg3AeabUFzkkmGwSjEAAMFGuAmw2jk35ZUelbiqTK4GAICWh3ATYJEOu6KddklMKgYAwAyEm0bAWjcAAJiHcNMIkgg3AACYhnDTCHzhhlswAAAQdISbRlC71g1zbgAACD7CTSOoXeuGcAMAQPARbhoBE4oBADAP4aYRHJlQzJwbAACCjXDTCI7cX4qRGwAAgo1w0whqT0sVlFXKVeUxuRoAAFoWwk0jiI8MU5jNIknKK3GbXA0AAC0L4aYRWCwWJUWz1g0AAGYg3DSSpFjm3QAAYAbCTSPhcnAAAMxBuGkkteGGhfwAAAguwk0jOXILBubcAAAQTISbRnLk5pmM3AAAEEyEm0bCnBsAAMxBuGkk3DwTAABzEG4aSe2cm7wSl7xew+RqAABoOQg3jaR1tEMWi1TlNZRfxirFAAAEC+GmkYTZrEqIdEhiUjEAAMFEuGlEviumuBwcAICgIdw0omRuwQAAQNARbhoRqxQDABB8hJtG5FvrhjuDAwAQNISbRpTCaSkAAIKOcNOIakduchi5AQAgaAg3jah2lWJGbgAACB7Tw83zzz+v9PR0hYeHKzMzUytXrjxh/2eeeUZdu3ZVRESE0tLSdOedd6qiommOjNSuUpxb7JJhsEoxAADBYGq4mTNnjqZOnarp06frm2++Ue/evTVs2DDl5uYes/+///1v3X333Zo+fbo2b96sl156SXPmzNGf//znIFdeN7Xr3LirvCosrzS5GgAAWgZTw83TTz+tG2+8URMnTtQZZ5yhWbNmKTIyUi+//PIx+3/11VcaPHiwfvWrXyk9PV0XX3yxxowZc9LRHrOEh9kUHxkmiVNTAAAEi2nhxu12a82aNcrKyjpSjNWqrKwsLV++/Jj7nHPOOVqzZo0vzOzYsUPz5s3TJZdcEpSa64NJxQAABJfdrA/Oy8uTx+NRSkqKX3tKSoq2bNlyzH1+9atfKS8vT0OGDJFhGKqqqtLNN998wtNSLpdLLteRUZOioqLAfAN1lBwTrm05JdxfCgCAIDF9QvGpWLp0qR555BG98MIL+uabb/Tee+/pk08+0UMPPXTcfWbMmKG4uDjflpaWFsSKj1wxlcP9pQAACArTRm4SExNls9mUk5Pj156Tk6PU1NRj7nPvvffq+uuv129/+1tJ0plnnqnS0lJNmjRJ06ZNk9V6dFa75557NHXqVN/zoqKioAYc3xVTjNwAABAUpo3cOBwO9evXT4sXL/a1eb1eLV68WIMGDTrmPmVlZUcFGJvNJknHvdTa6XQqNjbWbwumlFjuLwUAQDCZNnIjSVOnTtX48ePVv39/DRw4UM8884xKS0s1ceJESdK4cePUrl07zZgxQ5I0cuRIPf300+rbt68yMzO1fft23XvvvRo5cqQv5DQ1tSM3TCgGACA4TA03o0eP1sGDB3XfffcpOztbffr00YIFC3yTjPfs2eM3UvOXv/xFFotFf/nLX7Rv3z4lJSVp5MiRevjhh836Fk6KVYoBAAgui9HCls4tKipSXFycCgsLg3KKas+hMp33xBI57VZteWi4LBZLo38mAACh5lR+fzerq6Wao9qRG1eVV0UVVSZXAwBA6CPcNLLwMJtiw6vP/h3kcnAAABod4SYIkmNrJxUz7wYAgMZGuAmCFN+kYkZuAABobISbIDhyOTgjNwAANDbCTRDU3jyTVYoBAGh8hJsgqJ1zw2kpAAAaH+EmCBi5AQAgeAg3QZDCyA0AAEFDuAmC2pGbnCLXcW/wCQAAAoNwEwS1qxSXV3pU4mKVYgAAGhPhJggiHXbFOKtXKeYGmgAANC7CTZAkxdaemmLeDQAAjYlwEyQpNQv5HWTkBgCARkW4CZJkRm4AAAgKwk2QsNYNAADBQbgJkiNr3RBuAABoTISbIEmK4bQUAADBQLgJktqRGyYUAwDQuAg3QZLMyA0AAEFBuAmS2juDl7o9KmWVYgAAGg3hJkiinXZFOWySmFQMAEBjItwEUe3oDaemAABoPISbIPKtdcPIDQAAjYZwE0S1Ize5jNwAANBoCDdBlMLIDQAAjY5wE0S195di5AYAgMZDuAmi5JjaCcWM3AAA0FgIN0HkG7kpZuQGAIDGQrgJotqRG+4MDgBA4yHcBFGbuOpwU+yqUlFFpcnVAAAQmgg3QRTltCshyiFJ2ne43ORqAAAITYSbIGsXHyFJ+pFwAwBAoyDcBFn7VrXhpszkSgAACE2EmyCrDTeclgIAoHEQboKM01IAADQuwk2QtW8VKUn6sYDTUgAANAbCTZC1T+C0FAAAjYlwE2S1p6UOl1WqxFVlcjUAAIQewk2QxYSHKS4iTBKjNwAANAbCjQl8V0wx7wYAgIAj3JiAK6YAAGg8hBsT+K6YItwAABBwhBsTsEoxAACNh3BjgnasUgwAQKMh3JjgyMgN4QYAgEAj3Jigds7NoVK3ytysdQMAQCARbkwQFxGmmHC7JGl/AaM3AAAEEuHGJLWXg+/l1BQAAAFFuDEJl4MDANA4CDcm4XJwAAAaB+HGJO25HBwAgEZBuDEJl4MDANA4CDcmYc4NAACNg3BjktqrpfJKXKqo9JhcDQAAoYNwY5L4yDBFOWySpH2sdQMAQMAQbkxisVg4NQUAQCMg3JiIG2gCABB4hBsTsdYNAACBR7gxEZeDAwAQeIQbEx2Zc8PIDQAAgUK4MVHt5eBcLQUAQOAQbkxUe1oqp8glVxVr3QAAEAiEGxMlRDkUEVa91s3+ggqTqwEAIDQQbkxksVi4HBwAgAAzPdw8//zzSk9PV3h4uDIzM7Vy5coT9i8oKNCUKVPUpk0bOZ1OnX766Zo3b16Qqg08LgcHACCw7GZ++Jw5czR16lTNmjVLmZmZeuaZZzRs2DBt3bpVycnJR/V3u936xS9+oeTkZL377rtq166ddu/erfj4+OAXHyC14WZPPuEGAIBAMDXcPP3007rxxhs1ceJESdKsWbP0ySef6OWXX9bdd999VP+XX35Z+fn5+uqrrxQWFiZJSk9PD2bJAZeRFC1J+j63xORKAAAIDaadlnK73VqzZo2ysrKOFGO1KisrS8uXLz/mPh9++KEGDRqkKVOmKCUlRT179tQjjzwij+f4Vxq5XC4VFRX5bU1J19QYSdLW7GKTKwEAIDSYFm7y8vLk8XiUkpLi156SkqLs7Oxj7rNjxw69++678ng8mjdvnu6991499dRT+r//+7/jfs6MGTMUFxfn29LS0gL6fTRU15TqcLMnv0ylriqTqwEAoPkzfULxqfB6vUpOTtaLL76ofv36afTo0Zo2bZpmzZp13H3uueceFRYW+ra9e/cGseKTax3tVGK0UxKnpgAACATT5twkJibKZrMpJyfHrz0nJ0epqanH3KdNmzYKCwuTzWbztXXv3l3Z2dlyu91yOBxH7eN0OuV0OgNbfIB1S43Rsu0ubc0uUp+0eLPLAQCgWTNt5MbhcKhfv35avHixr83r9Wrx4sUaNGjQMfcZPHiwtm/fLq/X62vbtm2b2rRpc8xg01ycXnNqagvzbgAAaDBTT0tNnTpVs2fP1muvvabNmzdr8uTJKi0t9V09NW7cON1zzz2+/pMnT1Z+fr5uv/12bdu2TZ988okeeeQRTZkyxaxvISC6MakYAICAMfVS8NGjR+vgwYO67777lJ2drT59+mjBggW+ScZ79uyR1Xokf6WlpenTTz/VnXfeqV69eqldu3a6/fbb9ac//cmsbyEgaq+Y2pZDuAEAoKEshmEYZhcRTEVFRYqLi1NhYaFiY2PNLkeSVOauUo/pn8owpNV/yfJNMAYAANVO5fd3s7paKlRFOuzqkBApiVNTAAA0FOGmiahd74ZwAwBAwxBumggmFQMAEBiEmyaia2r1+cMtTCoGAKBBCDdNRNfUmhto5hTL621Rc7wBAAgowk0Tkd46Sg67VWVuj348XG52OQAANFuEmybCbrOqS1L16M2W7KZ153IAAJoTwk0T0pVJxQAANBjhpgnxhRsmFQMAUG+EmyaEkRsAABqOcNOE1K51syOvVK4qj8nVAADQPBFumpDU2HDFhNvl8RracbDU7HIAAGiWCDdNiMViYaViAAAaiHDTxNTOu9lCuAEAoF4IN01M7W0YtrLWDQAA9UK4aWJq7w6+LafE5EoAAGieCDdNTO1pqX0F5SosqzS5GgAAmp96hZu9e/fqxx9/9D1fuXKl7rjjDr344osBK6yliosIU6fEKEnS6t35JlcDAEDzU69w86tf/UpLliyRJGVnZ+sXv/iFVq5cqWnTpunBBx8MaIEtUWanBEnSip2EGwAATlW9ws3GjRs1cOBASdLbb7+tnj176quvvtKbb76pV199NZD1tUiZnWvCzY5DJlcCAEDzU69wU1lZKafTKUlatGiRLrvsMklSt27ddODAgcBV10JldmotSdq4v0glriqTqwEAoHmpV7jp0aOHZs2apf/9739auHChhg8fLknav3+/WrduHdACW6K28RFKS4iQx2to9S5OTQEAcCrqFW4ee+wx/eMf/9D555+vMWPGqHfv3pKkDz/80He6Cg1TO3qzknk3AACcEnt9djr//POVl5enoqIitWrVytc+adIkRUZGBqy4lmxgpwS9u+ZHJhUDAHCK6jVyU15eLpfL5Qs2u3fv1jPPPKOtW7cqOTk5oAW2VGfXjNx8+2OByt3cIRwAgLqqV7i5/PLL9frrr0uSCgoKlJmZqaeeekpXXHGFZs6cGdACW6q0hAi1iQtXpcfQN3sOm10OAADNRr3CzTfffKNzzz1XkvTuu+8qJSVFu3fv1uuvv65nn302oAW2VBaL5ch6N1wSDgBAndUr3JSVlSkmpvo2AZ999pmuuuoqWa1WnX322dq9e3dAC2zJMjtXn5r6mnk3AADUWb3CTZcuXTR37lzt3btXn376qS6++GJJUm5urmJjYwNaYEtWO3Kzbm+BKiqZdwMAQF3UK9zcd999uuuuu5Senq6BAwdq0KBBkqpHcfr27RvQAluyTolRSopxyl3l1bq9BWaXAwBAs1CvcHP11Vdrz549Wr16tT799FNf+0UXXaS//vWvASuupfOfd8OpKQAA6qJe69xIUmpqqlJTU313B2/fvj0L+DWCzE4J+vjbA1q565Ck08wuBwCAJq9eIzder1cPPvig4uLi1LFjR3Xs2FHx8fF66KGH5PV6A11ji1Y7qXjN7sNyV3FsAQA4mXqN3EybNk0vvfSSHn30UQ0ePFiStGzZMt1///2qqKjQww8/HNAiW7LTkqOVEOVQfqlbG/YVqF/HBLNLAgCgSatXuHnttdf0z3/+03c3cEnq1auX2rVrp1tuuYVwE0AWi0UD0xO0YFO2vt6RT7gBAOAk6nVaKj8/X926dTuqvVu3bsrPZ+JroA3KqD419d+tB02uBACApq9e4aZ379567rnnjmp/7rnn1KtXrwYXBX9ZZ6RIklbtzlducYXJ1QAA0LTV67TU448/rksvvVSLFi3yrXGzfPly7d27V/PmzQtogZDaxUeoT1q81u0t0KebcnT92R3NLgkAgCarXiM3Q4cO1bZt23TllVeqoKBABQUFuuqqq7Rp0ya98cYbga4Rkkb0TJUkzd9wwORKAABo2iyGYRiBerP169frrLPOksfTdG8VUFRUpLi4OBUWFjarW0XsOVSm855YIpvVopV/vkito51mlwQAQNCcyu/veo3cIPg6tI5Uj7ax8ngNLfwux+xyAABosgg3zcglZ7aRJM3fmG1yJQAANF2Em2akdt7Nl9vzVFhWaXI1AAA0Tad0tdRVV111wtcLCgoaUgtOonNStLqmxGhrTrEWbc7RqH7tzS4JAIAm55TCTVxc3ElfHzduXIMKwomNODNVW3OKNX/jAcINAADHcErh5pVXXmmsOlBHI3q20TOLvtcX2/JUXFGpmPAws0sCAKBJYc5NM3N6SrQ6J0XJ7fHq8y25ZpcDAECTQ7hpZiwWiy7pWXPV1AaumgIA4OcIN83Q8JqrppZuy1WZu8rkagAAaFoIN81Qj7axSm8dqYpKrz75ltsxAADwU4SbZshisejaAWmSpDdX7DG5GgAAmhbCTTN1Tb80hdksWre3QJv2F5pdDgAATQbhpplKinHq4h7Vc2/+zegNAAA+hJtmbGxmB0nS3LX7VOJiYjEAABLhplkb1Lm1OidGqdTt0Yfr9ptdDgAATQLhphmzWCz6Vc3ozZsrdsswDJMrAgDAfISbZm7UWe3lsFu1aX+R1v/IxGIAAAg3zVyrKIcuPbN6xeJ/r9htcjUAAJiPcBMCaicWf7h+vwrLK02uBgAAcxFuQkC/jq10ekq0Kiq9mrt2n9nlAABgKsJNCLBYLBqb2VGS9OpXu+TxMrEYANByEW5CxKh+7RUXEaadeaVasJG7hQMAWi7CTYiIdto1/px0SdILS7dzWTgAoMUi3ISQieekKyLMpk37i/TfbQfNLgcAAFMQbkJIqyiHb1G/F5b+YHI1AACYo0mEm+eff17p6ekKDw9XZmamVq5cWaf93nrrLVksFl1xxRWNW2Az8ttzOynMZtHKnflavSvf7HIAAAg608PNnDlzNHXqVE2fPl3ffPONevfurWHDhik3N/eE++3atUt33XWXzj333CBV2jy0iYvQqLPaS2L0BgDQMpkebp5++mndeOONmjhxos444wzNmjVLkZGRevnll4+7j8fj0dixY/XAAw+oc+fOQay2ebhpaIasFunzLbnafKDI7HIAAAgqU8ON2+3WmjVrlJWV5WuzWq3KysrS8uXLj7vfgw8+qOTkZP3mN7856We4XC4VFRX5baGuU2KURtTckmEmozcAgBbG1HCTl5cnj8ejlJQUv/aUlBRlZx97rZZly5bppZde0uzZs+v0GTNmzFBcXJxvS0tLa3DdzcEt52dIkj7+dr925pWaXA0AAMFj+mmpU1FcXKzrr79es2fPVmJiYp32ueeee1RYWOjb9u7d28hVNg092sbpwm7J8hrSM4u2mV0OAABBYzfzwxMTE2Wz2ZSTk+PXnpOTo9TU1KP6//DDD9q1a5dGjhzpa/N6vZIku92urVu3KiMjw28fp9Mpp9PZCNU3fVN/cbo+35KrD9fv103nZeiMtrFmlwQAQKMzdeTG4XCoX79+Wrx4sa/N6/Vq8eLFGjRo0FH9u3Xrpg0bNmjdunW+7bLLLtMFF1ygdevWtZhTTnXVs12cftmrjQxDevKzrWaXAwBAUJg6ciNJU6dO1fjx49W/f38NHDhQzzzzjEpLSzVx4kRJ0rhx49SuXTvNmDFD4eHh6tmzp9/+8fHxknRUO6r9/uKumr8xW59vydXqXfnqn55gdkkAADQq08PN6NGjdfDgQd13333Kzs5Wnz59tGDBAt8k4z179shqbVZTg5qUTolRurZ/e/1n5V49tmCL3r5pkCwWi9llAQDQaCxGC7vDYlFRkeLi4lRYWKjY2JYxB+VAYbnOf2KpXFVevTJhgC7olmx2SQAAnJJT+f3NkEgL0CYuwnfH8Mc/3Sqvt0XlWQBAC0O4aSEmD81QjNOuzQeK9PGGA2aXAwBAoyHctBCtohyadF71rSqe/HSrXFUekysCAKBxEG5akBuGdFJyjFN78sv06pe7zC4HAIBGQbhpQaKcdv1xeDdJ0t8/366DxS6TKwIAIPAINy3MVX3bqVf7OJW4qvQUC/sBAEIQ4aaFsVotuu+XZ0iS5qzeq037C02uCACAwCLctED90xM0sndbGYb04EffqYUtdQQACHGEmxbq7hHd5LRbtWJnvhZszDa7HAAAAoZw00K1i4/QTTWXhj8yf7MqKrk0HAAQGgg3LdjN52coJdapvfnlmv3FDrPLAQAgIAg3LVikw64/X9JdkvTcku3ac6jM5IoAAGg4wk0Ld1nvthrcpbVcVV5N/3Ajk4sBAM0e4aaFs1gsevDyngqzWbRk60F9uinH7JIAAGgQwg2UkRStm87LkCQ98NEmlbqqTK4IAID6I9xAknTrhV2UlhChA4UV+tvi780uBwCAeiPcQJIUHmbTg5f1lCS9tGyntmYXm1wRAAD1Q7iBzwXdkjWsR4o8XkPT3t8gr5fJxQCA5odwAz/TR/ZQpMOm1bsP682Ve8wuBwCAU0a4gZ+28RH60/BukqRH523W/oJykysCAODUEG5wlOvP7qh+HVup1O3RX+ay9g0AoHkh3OAoVqtFj406Uw6bVZ9vydWH6/ebXRIAAHVGuMExdUmO0e8u6iJJeuCj73SoxGVyRQAA1A3hBsd109AMdUuNUX6pWw9+/J3Z5QAAUCeEGxxXmM2qx6/uJatF+mDdfi36jlszAACaPsINTqhX+3jdeG5nSdI972/Q4VK3yRUBAHBihBuc1J2/OF2nJUfrYLGLq6cAAE0e4QYnFR5m09PX9pHdatEnGw5w9RQAoEkj3KBOzmwfp99ddJok6d65G5VdWGFyRQAAHBvhBnV2y/kZ6t0+TkUVVfrDu+s5PQUAaJIIN6gzu82qp67tI6fdqv99n6d/reDeUwCApodwg1PSJTlad4+ovvfUI59s1vbcEpMrAgDAH+EGp2z8oHQN6ZKo8kqPbvvPWlVUeswuCQAAH8INTpnVatHT1/ZW6yiHNh8o0qPzt5hdEgAAPoQb1EtybLievLa3JOnVr3ZpIasXAwCaCMIN6u2Crsn67ZBOkqQ/vLteBwrLTa4IAADCDRroj8O76cx2cSooq9Qdb62Tx8vl4QAAcxFu0CAOu1V/H9NXUQ6bVuzM17OLvze7JABAC0e4QYOlJ0bp4SvPlCQ9+/n3Wro11+SKAAAtGeEGAXFF33Yam9lBhiHd/tY67c0vM7skAEALRbhBwNw38gz1TotXYXmlJr+5hvVvAACmINwgYJx2m14Ye5YSohzauK9I0z/YZHZJAIAWiHCDgGoXH6Fnr+srq0Was3qv5qzi/lMAgOAi3CDghpyWqN9f3FWSdO8Hm7Rub4G5BQEAWhTCDRrF5KEZ+sUZKXJXeXXj66tZ4A8AEDSEGzQKq9Wiv47uo26pMTpY7NKNr69WmbvK7LIAAC0A4QaNJtpp1+xx/dW6ZoLxXe+sl5cVjAEAjYxwg0aVlhCpWdf3U5jNonkbsvXMom1mlwQACHGEGzS6AekJesS3gvF2fbBun8kVAQBCGeEGQXFN/zRNOq+zJOkP73yrr37IM7kiAECoItwgaP40vJsuOTNVbo9Xk15fo037C80uCQAQggg3CBqb1aKnr+2jzE4JKnFVacIrq7gHFQAg4Ag3CKrwMJteHNffd4n4uJdX6lCJy+yyAAAhhHCDoIuLCNNrNwxUu/gI7cwr1Q2vrVapizVwAACBQbiBKVJiw/XaDQPVKjJM6/cW6IZXV6nczV3EAQANR7iBabokR+vViQMV47Rrxc583fj6alVUEnAAAA1DuIGpeqfF69UbBijSYdOy7Xm6+V9r5Koi4AAA6o9wA9P165igVyYMUHiYVUu3HtSt/16rSo/X7LIAAM0U4QZNQmbn1vrnuAFy2K1a+F2Obvv3WrmrCDgAgFNHuEGTMeS0RL14fT85bFYt2JStm95gDg4A4NQRbtCknN81Wf8c31/hYVYt2XpQE19ZxWXiAIBTQrhBk3Pe6Ul6/YZMRTvtWr7jkK5/aYUKyyvNLgsA0EwQbtAkDeyUoH/9NlNxEWH6Zk+BfjX7a+WxkjEAoA4IN2iy+qTF661JZysx2qFN+4t05QtfantuidllAQCaOMINmrTubWL19k2D1CEhUnvzyzVq5lf6eschs8sCADRhTSLcPP/880pPT1d4eLgyMzO1cuXK4/adPXu2zj33XLVq1UqtWrVSVlbWCfuj+eucFK33bzlHfTvEq7C8Ute/tELvr/3R7LIAAE2U6eFmzpw5mjp1qqZPn65vvvlGvXv31rBhw5Sbm3vM/kuXLtWYMWO0ZMkSLV++XGlpabr44ou1b9++IFeOYGod7dR/bjxbI3qmqtJj6M456/XMom3yeg2zSwMANDEWwzBM/e2QmZmpAQMG6LnnnpMkeb1epaWl6bbbbtPdd9990v09Ho9atWql5557TuPGjTtp/6KiIsXFxamwsFCxsbENrh/B5fUaemzBFv3jix2SpIvPSNHTo/so2mk3uTIAQGM6ld/fpo7cuN1urVmzRllZWb42q9WqrKwsLV++vE7vUVZWpsrKSiUkJBzzdZfLpaKiIr8NzZfVatE9l3TXY6POlMNm1Wff5eiK57/UDweZaAwAqGZquMnLy5PH41FKSopfe0pKirKzs+v0Hn/605/Utm1bv4D0UzNmzFBcXJxvS0tLa3DdMN/oAR0056azlRLr1PbcEl3x3Jda9F2O2WUBAJoA0+fcNMSjjz6qt956S++//77Cw8OP2eeee+5RYWGhb9u7d2+Qq0Rj6duhlT66bYgGpLdSsatKv319tZ74dIuquOkmALRopoabxMRE2Ww25eT4/487JydHqampJ9z3ySef1KOPPqrPPvtMvXr1Om4/p9Op2NhYvw2hIzkmXG/+9myNG9RRkvT8kh80ZvbX2l9QbnJlAACzmBpuHA6H+vXrp8WLF/vavF6vFi9erEGDBh13v8cff1wPPfSQFixYoP79+wejVDRhDrtVD17eU38f01fRTrtW7TqsS579H6epAKCFMv201NSpUzV79my99tpr2rx5syZPnqzS0lJNnDhRkjRu3Djdc889vv6PPfaY7r33Xr388stKT09Xdna2srOzVVLChNKWbmTvtvrkd0N0Zrs4FZRV6revr9b9H25SuZs7iwNAS2J6uBk9erSefPJJ3XffferTp4/WrVunBQsW+CYZ79mzRwcOHPD1nzlzptxut66++mq1adPGtz355JNmfQtoQjq2jtK7kwfphsGdJEmvfrVLlzz7P63elW9yZQCAYDF9nZtgY52blmPJllzd/d63yilyyWKRbhjcSXdd3FURDpvZpQEATlGzWecGaEwXdEvWZ3cM1dX92sswpJeW7dSIv33BvakAIMQRbhDS4iLD9OQ1vfXKhAFKjQ3XrkNluu7Fr3XXO+uVX+o2uzwAQCMg3KBFuKBbsj698zz9KrODJOndNT/qoqeW6u3Ve9XCzswCQMgj3KDFiIsI0yNXnqn/N/kcdUuN0eGySv3x3W81+h9fa+O+QrPLAwAECBOK0SJVerx6edlOPbPoe5VXemSxSFef1V5/GNZVybHHXu0aAGCeU/n9TbhBi7avoFyPzd+iD9fvlyRFOmyaPDRDN57XWeFhXFUFAE0F4eYECDc4ljW7D+uhj7/Tur0FkqSUWKduu/A0Xds/TQ47Z28BwGyEmxMg3OB4vF5DH327X48v2Kp9Nfem6pAQqTuyTtPlfdrJZrWYXCEAtFyEmxMg3OBkXFUe/WfFHj235AfllbgkSaclR2vKBV30y15tZLcxkgMAwUa4OQHCDeqqzF2lV7/apX/8d4cKyyslSWkJEbrpvAxd3a89c3IAIIgINydAuMGpKqqo1BvLd+vlZTt1qGbhv6QYpyYOTtevBnZQfKTD5AoBIPQRbk6AcIP6Knd7NGfVHr34xQ7tL6yQJEWE2XR1v/aaODhdnZOiTa4QAEIX4eYECDdoKHeVVx+u369//m+HtmQXS5IsFunCrsm6flBHnXdakqxMPgaAgCLcnADhBoFiGIaW/3BILy3bqcVbcn3taQkRGjOwg67pl6akGKeJFQJA6CDcnADhBo1hx8ES/evrPXp3zV4VVVRJksJsFl18Rqqu7t9e552WxKXkANAAhJsTINygMZW7PfpkwwG9uWK31u4p8LWnxobrqrPa6Zr+aeqUGGVegQDQTBFuToBwg2D5bn+R3l69Vx+s26fDZZW+9t5p8RrZq41+2autUuO4jxUA1AXh5gQINwg2V5VHizfn6u3Ve/XFtoPy1vyNs1ikgekJ+mWvNhrWI5UbdgLACRBuToBwAzPlFldo/oZsfbR+v1bvPuxrt1ikszq00vAeqRrWI1UdWkeaWCUAND2EmxMg3KCp2FdQrk++3a/5G7P95udIUrfUGGV1T1HWGSnq1S6OS8sBtHiEmxMg3KApyi6s0GffZWvBxmyt2Jkvj/fIX8ukGKcu7JqsoV2TNLhLouIiwkysFADMQbg5AcINmrrDpW4t3ZarRd/l6r/bDqrEVeV7zWa1qE9avIaenqQhpyWqV7s4buQJoEUg3JwA4QbNibvKqxU7D2np1oP677aD2p5b4vd6tNOuzE4JGpTRWudkJKpbagynsACEJMLNCRBu0Jz9eLhMX2zL0xfbDmr5jkO+u5XXig23a0B6ggZ0StDATgnq2TZODjsjOwCaP8LNCRBuECo8XkObDxTpqx/y9OX2Q1q9K1+lbo9fH6fdqt7t43VWx1bq17GVzuoQr9bR3BICQPNDuDkBwg1CVZXHq+8OFGnlznyt3JmvVbvy/RYPrNUhIVK90+LVJy1efdLi1KNtnMLDbCZUDAB1R7g5AcINWgrDMLQjr1Rrdh/WN7sPa83uw/r+Z3N2pOpJyqclR6tnuzid2S5OPdvFqltqrKKcdhOqBoBjI9ycAOEGLVlheaU2/FiodXsPa93eQq3bW6C8EtdR/SwWqVPrKHVvG6sz2lRvp6fGqG1cuCwWJiwDCD7CzQkQboAjDMNQTpFLG/YVasO+Qm2q+ZpbfHTgkaSYcLu6psSoa2qMTk+J0WnJ0eqSEq2kaCehB0CjItycAOEGOLm8Epc2HyjSd/uLtGl/kbZmF+uHgyWq8h77n4u4iDCdlhytjKRoZSRHKSMpWp2TopXWKoJ1eAAEBOHmBAg3QP24q7zakVeirdnF2pJdrO25JdqeW6Ldh0p1nMyjMJtFaQmR6pwYrc5JUUpvHaWOrSPVISFSbeMjZGNNHgB1RLg5AcINEFgVlR7tOFiqHw6W1Gyl+iG3RDvySlRR6T3ufmE2i9q3ilT7VhFKS6j5+pPnraMcnOoC4HMqv7+5HAJAg4SH2XRG21id0db/Hxuv11B2UYV2HCzVzrwS7cgr1c68Uu3JL9OP+eVye7zaWdN2LBFhNr/g0zY+Qu3ij3xNinEy8gPgmBi5ARB0nprgs/tQqX7ML9ePh8u093C59uaXae/hMuUWu3Syf5nsVotSYsPVJi5cqXHVX1Nif7o5lRwTrggHa/gAoYCRGwBNms1qUbuaERhlHP26q8qj/QUVvrCzv6Bc+w6Xa39BhfYVlCu7qEJVXkP7Csq1r6D8hJ8VE25Xckx10EmOdfo9Toqpfp4UHa7YCDunwYAQQbgB0OQ47TZ1SoxSp8SoY75e5fHqYIlLBwordKCgQgcKy5VdWKGcYpdyiip8W0WlV8UVVSquqNIPB499+quWw2ZVUoxTidEOtY52qnXUT7/6P06IcshpZ0QIaKoINwCaHbvNqjZxEWoTFyF1OHYfwzBU7KpSbpFLucUVOljs8j3O/cnjg8UuFVVUye3x1mkkqFa0065WUWFKiHSoVZRDrSIdiosIU6tIh+IjwxQfWf24VaRDraKqH0c6bIwOAUFAuAEQkiwWi2LDwxQbHqYuydEn7FtR6VFeiUsHi6u3/FK3DpW6lVfi0qESt/JrH5dWP/Z4DZW4qlTiqtLe/LqFIal6dCg2wq7YiOq64iL8t/jIMMVGHN0eGxGmKIIRUGeEGwAtXniYreay9MiT9vV6DRVVVCq/1K3DZW7ll1bqcKlbBeVuHS6rVEFZpQrL3TpcWqnDZe6arVLuKq/cHq/yStzKK3Gfco1WixQTHqaYcLvva2zN42in3dceXdMe7azZwu2KcVa3RzvtcthZVBGhj3ADAKfAarUoPtKh+EhHnfcxDENlbo8KyitVVF6pwp9sP31eHYyObq/yGvIa8j2X6j5a9HMOm1VRTpuiasJPVO3m+GmbTdHOMEU7bYoOtysizK5Ih02RDpsiHDZFOuy+xxFhNoWxCjWaGMINADQyi8XiCxHt4iNOaV/DMGomRleqqKJSheVVKq6oVImreqJ0SUX18+KfPndVVj/+SVt5pUeS5PZ45S7z6nBZZcC+vzCbRRFh1eEowmFTlMNeE4Jqt5owFGZTeM0WEWZVhKP6caTDrogwmyIc1prXjgSn8DCbnHYrp+RwSgg3ANCEWSyW6l/0DpuSY8Pr/T5VHq9KXR6VuKtUWjNfqKSiSmXuKpW6PCp1V7eVuTy+UFTbr8xdpTK3R+WVnuqvbo/K3FW+225UegxVeqpUVFEVoO/6aE67VU679SfhyKZwR3VICg+zKdxukzPMWtPPpvCwI1/Dw2xyhtkUbrfKWROWavv9dB+n3VrzvPqxw2aVlYUimyXCDQC0AHabVXGRVsVFhgXk/QzDkNvjrQk6R0JPqbvK9/VIEKoOQxWV1QGp3O1VRVX1axU1gemnX8srq79Weo6s5Oiq8spV5W3UAHUsDptVDl8Yqn7s23yv1QYjm38/m1VhNqvsNovCbNXtYbaf73vkPXz72Kv71+4fZrPI7ntukc1qYSTrJAg3AIBTZrFYan6p2xR/8nnY9VLp8aqi0iNXlf/Xisrqr+U1o0nlNa+5ar/WPPb1rfL49ndVVk/sdlVVv+b+2Xu7PV6/1bHdnur+Ja7G+R7rw2LRT8KPpSYAVYekMJtFdqtVYXarHDZLTZu1uq0mJIXZLAqzVocou/XIfrXvY7f+pF/N8yPvU91ms1pkr/ksu83iF+QctupTjonRTtOOEeEGANAk1f6yjQniZxqGoSqv4ReW3DWByV1VHZTcP23z1D72yFXp9YWkSo+3ZjNUWdOn0lPbv3rUy1UTpmrfy1XlUWWVoSpvbf/qfau8xs9qlK+GpqpPWrzmThls2ucTbgAAqGGxWHwjFtHOpvEr0jAMX9CpDUxuj1eVNYHJVVUdgKo8R8JWVW2oqu1f5VWVt/pxVU1g8nu/mveq8hiq9FZ/rQ5ZRs1+PwlbNV89XsP3Pj/9vCqPofAwc6+gaxo/OQAAcEwWi0UOu4U1ik4BRwoAAIQUwg0AAAgphBsAABBSCDcAACCkEG4AAEBIIdwAAICQQrgBAAAhhXADAABCCuEGAACEFMINAAAIKYQbAAAQUgg3AAAgpBBuAABASCHcAACAkGI3u4BgMwxDklRUVGRyJQAAoK5qf2/X/h4/kRYXboqLiyVJaWlpJlcCAABOVXFxseLi4k7Yx2LUJQKFEK/Xq/379ysmJkYWiyWg711UVKS0tDTt3btXsbGxAX1v+ONYBw/HOng41sHDsQ6eQB1rwzBUXFystm3bymo98ayaFjdyY7Va1b59+0b9jNjYWP6yBAnHOng41sHDsQ4ejnXwBOJYn2zEphYTigEAQEgh3AAAgJBCuAkgp9Op6dOny+l0ml1KyONYBw/HOng41sHDsQ4eM451i5tQDAAAQhsjNwAAIKQQbgAAQEgh3AAAgJBCuAEAACGFcBMgzz//vNLT0xUeHq7MzEytXLnS7JKavRkzZmjAgAGKiYlRcnKyrrjiCm3dutWvT0VFhaZMmaLWrVsrOjpao0aNUk5OjkkVh45HH31UFotFd9xxh6+NYx04+/bt069//Wu1bt1aEREROvPMM7V69Wrf64Zh6L777lObNm0UERGhrKwsff/99yZW3Dx5PB7de++96tSpkyIiIpSRkaGHHnrI795EHOv6++KLLzRy5Ei1bdtWFotFc+fO9Xu9Lsc2Pz9fY8eOVWxsrOLj4/Wb3/xGJSUlDS/OQIO99dZbhsPhMF5++WVj06ZNxo033mjEx8cbOTk5ZpfWrA0bNsx45ZVXjI0bNxrr1q0zLrnkEqNDhw5GSUmJr8/NN99spKWlGYsXLzZWr15tnH322cY555xjYtXN38qVK4309HSjV69exu233+5r51gHRn5+vtGxY0djwoQJxooVK4wdO3YYn376qbF9+3Zfn0cffdSIi4sz5s6da6xfv9647LLLjE6dOhnl5eUmVt78PPzww0br1q2Njz/+2Ni5c6fxzjvvGNHR0cbf/vY3Xx+Odf3NmzfPmDZtmvHee+8Zkoz333/f7/W6HNvhw4cbvXv3Nr7++mvjf//7n9GlSxdjzJgxDa6NcBMAAwcONKZMmeJ77vF4jLZt2xozZswwsarQk5uba0gy/vvf/xqGYRgFBQVGWFiY8c477/j6bN682ZBkLF++3Kwym7Xi4mLjtNNOMxYuXGgMHTrUF2441oHzpz/9yRgyZMhxX/d6vUZqaqrxxBNP+NoKCgoMp9Np/Oc//wlGiSHj0ksvNW644Qa/tquuusoYO3asYRgc60D6ebipy7H97rvvDEnGqlWrfH3mz59vWCwWY9++fQ2qh9NSDeR2u7VmzRplZWX52qxWq7KysrR8+XITKws9hYWFkqSEhARJ0po1a1RZWel37Lt166YOHTpw7OtpypQpuvTSS/2OqcSxDqQPP/xQ/fv31zXXXKPk5GT17dtXs2fP9r2+c+dOZWdn+x3ruLg4ZWZmcqxP0TnnnKPFixdr27ZtkqT169dr2bJlGjFihCSOdWOqy7Fdvny54uPj1b9/f1+frKwsWa1WrVixokGf3+JunBloeXl58ng8SklJ8WtPSUnRli1bTKoq9Hi9Xt1xxx0aPHiwevbsKUnKzs6Ww+FQfHy8X9+UlBRlZ2ebUGXz9tZbb+mbb77RqlWrjnqNYx04O3bs0MyZMzV16lT9+c9/1qpVq/S73/1ODodD48eP9x3PY/2bwrE+NXfffbeKiorUrVs32Ww2eTwePfzwwxo7dqwkcawbUV2ObXZ2tpKTk/1et9vtSkhIaPDxJ9ygWZgyZYo2btyoZcuWmV1KSNq7d69uv/12LVy4UOHh4WaXE9K8Xq/69++vRx55RJLUt29fbdy4UbNmzdL48eNNri60vP3223rzzTf173//Wz169NC6det0xx13qG3bthzrEMdpqQZKTEyUzWY76qqRnJwcpaammlRVaLn11lv18ccfa8mSJWrfvr2vPTU1VW63WwUFBX79Ofanbs2aNcrNzdVZZ50lu90uu92u//73v3r22Wdlt9uVkpLCsQ6QNm3a6IwzzvBr6969u/bs2SNJvuPJvykN94c//EF33323rrvuOp155pm6/vrrdeedd2rGjBmSONaNqS7HNjU1Vbm5uX6vV1VVKT8/v8HHn3DTQA6HQ/369dPixYt9bV6vV4sXL9agQYNMrKz5MwxDt956q95//319/vnn6tSpk9/r/fr1U1hYmN+x37p1q/bs2cOxP0UXXXSRNmzYoHXr1vm2/v37a+zYsb7HHOvAGDx48FFLGmzbtk0dO3aUJHXq1Empqal+x7qoqEgrVqzgWJ+isrIyWa3+v+ZsNpu8Xq8kjnVjqsuxHTRokAoKCrRmzRpfn88//1xer1eZmZkNK6BB05FhGEb1peBOp9N49dVXje+++86YNGmSER8fb2RnZ5tdWrM2efJkIy4uzli6dKlx4MAB31ZWVubrc/PNNxsdOnQwPv/8c2P16tXGoEGDjEGDBplYdej46dVShsGxDpSVK1cadrvdePjhh43vv//eePPNN43IyEjjX//6l6/Po48+asTHxxsffPCB8e233xqXX345lyfXw/jx44127dr5LgV/7733jMTEROOPf/yjrw/Huv6Ki4uNtWvXGmvXrjUkGU8//bSxdu1aY/fu3YZh1O3YDh8+3Ojbt6+xYsUKY9myZcZpp53GpeBNyd///nejQ4cOhsPhMAYOHGh8/fXXZpfU7Ek65vbKK6/4+pSXlxu33HKL0apVKyMyMtK48sorjQMHDphXdAj5ebjhWAfORx99ZPTs2dNwOp1Gt27djBdffNHvda/Xa9x7771GSkqK4XQ6jYsuusjYunWrSdU2X0VFRcbtt99udOjQwQgPDzc6d+5sTJs2zXC5XL4+HOv6W7JkyTH/jR4/frxhGHU7tocOHTLGjBljREdHG7GxscbEiRON4uLiBtdmMYyfLNUIAADQzDHnBgAAhBTCDQAACCmEGwAAEFIINwAAIKQQbgAAQEgh3AAAgJBCuAEAACGFcAOgxbNYLJo7d67ZZQAIEMINAFNNmDBBFovlqG348OFmlwagmbKbXQAADB8+XK+88opfm9PpNKkaAM0dIzcATOd0OpWamuq3tWrVSlL1KaOZM2dqxIgRioiIUOfOnfXuu+/67b9hwwZdeOGFioiIUOvWrTVp0iSVlJT49Xn55ZfVo0cPOZ1OtWnTRrfeeqvf63l5ebryyisVGRmp0047TR9++GHjftMAGg3hBkCTd++992rUqFFav369xo4dq+uuu06bN2+WJJWWlmrYsGFq1aqVVq1apXfeeUeLFi3yCy8zZ87UlClTNGnSJG3YsEEffvihunTp4vcZDzzwgK699lp9++23uuSSSzR27Fjl5+cH9fsEECANvvUmADTA+PHjDZvNZkRFRfltDz/8sGEY1XeHv/nmm/32yczMNCZPnmwYhmG8+OKLRqtWrYySkhLf65988olhtVqN7OxswzAMo23btsa0adOOW4Mk4y9/+YvveUlJiSHJmD9/fsC+TwDBw5wbAKa74IILNHPmTL+2hIQE3+NBgwb5vTZo0CCtW7dOkrR582b17t1bUVFRvtcHDx4sr9errVu3ymKxaP/+/broootOWEOvXr18j6OiohQbG6vc3Nz6fksATES4AWC6qKioo04TBUpERESd+oWFhfk9t1gs8nq9jVESgEbGnBsATd7XX3991PPu3btLkrp3767169ertLTU9/qXX34pq9Wqrl27KiYmRunp6Vq8eHFQawZgHkZuAJjO5XIpOzvbr81utysxMVGS9M4776h///4aMmSI3nzzTa1cuVIvvfSSJGns2LGaPn26xo8fr/vvv18HDx7Ubbfdpuuvv14pKSmSpPvvv18333yzkpOTNWLECBUXF+vLL7/UbbfdFtxvFEBQEG4AmG7BggVq06aNX1vXrl21ZcsWSdVXMr311lu65ZZb1KZNG/3nP//RGWecIUmKjIzUp59+qttvv10DBgxQZGSkRo0apaefftr3XuPHj1dFRYX++te/6q677lJiYqKuvvrq4H2DAILKYhiGYXYRAHA8FotF77//vq644gqzSwHQTDDnBgAAhBTCDQAACCnMuQHQpHHmHMCpYuQGAACEFMINAAAIKYQbAAAQUgg3AAAgpBBuAABASCHcAACAkEK4AQAAIYVwAwAAQgrhBgAAhJT/D5h6v49xYP74AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 设置GPU设备\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(f\"使用设备: {device}\")\n",
    "\n",
    "# 加载鸢尾花数据集\n",
    "iris = load_iris()\n",
    "X = iris.data  # 特征数据\n",
    "y = iris.target  # 标签数据\n",
    "\n",
    "# 划分训练集和测试集\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "# 归一化数据\n",
    "scaler = MinMaxScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)\n",
    "\n",
    "# 将数据转换为PyTorch张量并移至GPU\n",
    "X_train = torch.FloatTensor(X_train).to(device)\n",
    "y_train = torch.LongTensor(y_train).to(device)\n",
    "X_test = torch.FloatTensor(X_test).to(device)\n",
    "y_test = torch.LongTensor(y_test).to(device)\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(MLP, self).__init__()\n",
    "        self.fc1 = nn.Linear(4, 10)  # 输入层到隐藏层\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(10, 3)  # 隐藏层到输出层\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.fc1(x)\n",
    "        out = self.relu(out)\n",
    "        out = self.fc2(out)\n",
    "        return out\n",
    "\n",
    "# 实例化模型并移至GPU\n",
    "model = MLP().to(device)\n",
    "\n",
    "# 分类问题使用交叉熵损失函数\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# 使用随机梯度下降优化器\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)\n",
    "\n",
    "# 训练模型\n",
    "num_epochs = 20000  # 训练的轮数\n",
    "\n",
    "# 用于存储每100个epoch的损失值和对应的epoch数\n",
    "losses = []\n",
    "\n",
    "start_time = time.time()  # 记录开始时间\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    # 前向传播\n",
    "    outputs = model(X_train)  # 隐式调用forward函数\n",
    "    loss = criterion(outputs, y_train)\n",
    "\n",
    "    # 反向传播和优化\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 记录损失值\n",
    "    if (epoch + 1) % 200 == 0:\n",
    "        losses.append(loss.item()) # item()方法返回一个Python数值，loss是一个标量张量\n",
    "        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')\n",
    "    \n",
    "    # 打印训练信息\n",
    "    if (epoch + 1) % 100 == 0: # range是从0开始，所以epoch+1是从当前epoch开始，每100个epoch打印一次\n",
    "        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')\n",
    "\n",
    "time_all = time.time() - start_time  # 计算训练时间\n",
    "print(f'Training time: {time_all:.2f} seconds')\n",
    "\n",
    "\n",
    "# 可视化损失曲线\n",
    "plt.plot(range(len(losses)), losses)\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.title('Training Loss over Epochs')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f76dabee",
   "metadata": {},
   "source": [
    "这里我们定义剩余时长 = 总时长-必须的计算时长3s，可以多做几次实验，来对比下记录次数和剩余时长的分布关系，很容易以为这二者是成正比的\n",
    "\n",
    "下表为我在我的本地电脑上的测试结果，总epoch = 20000\n",
    "\n",
    "| 记录间隔（轮） | 记录次数（次） | 剩余时长（秒） |\n",
    "|----------------|----------------|----------------|\n",
    "| 100            | 200            | 10.43          |\n",
    "| 200            | 100            | 10.02          |\n",
    "| 1000           | 20             | 10.12          |\n",
    "| 2000           | 10             | 9.74           |\n",
    "\n",
    "\n",
    "可以发现，记录次数和剩余时长之间并无明显的线性关系。思考下为什么？\n",
    "\n",
    "我目前的理解是loss.item()是一个同步操作，gou需要等待cpu完成才能开启下次运算。但是仍然无法解释为为什么剩余时长和记录次数之间没有线性关系。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2a28dfc",
   "metadata": {},
   "source": [
    "## __call__方法\n",
    "\n",
    "在 Python 中，__call__ 方法是一个特殊的魔术方法（双下划线方法），它允许类的实例像函数一样被调用。这种特性使得对象可以表现得像函数，同时保留对象的内部状态。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a43092a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 我们来看下昨天代码中你的定义函数的部分\n",
    "class MLP(nn.Module): # 定义一个多层感知机（MLP）模型，继承父类nn.Module\n",
    "    def __init__(self): # 初始化函数\n",
    "        super(MLP, self).__init__() # 调用父类的初始化函数\n",
    " # 前三行是八股文，后面的是自定义的\n",
    "\n",
    "        self.fc1 = nn.Linear(4, 10)  # 输入层到隐藏层\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(10, 3)  # 隐藏层到输出层\n",
    "# 输出层不需要激活函数，因为后面会用到交叉熵函数cross_entropy，交叉熵函数内部有softmax函数，会把输出转化为概率\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.fc1(x)\n",
    "        out = self.relu(out)\n",
    "        out = self.fc2(out)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6cd987d",
   "metadata": {},
   "source": [
    "可以注意到，self.fc1 = nn.Linear(4, 10)  此时，是实例化了一个nn.Linear(4, 10)对象，并把这个对象赋值给了MLP的初始化函数中的self.fc1变量。\n",
    "\n",
    "那为什么下面的前向传播中却可以out = self.fc1(x)  呢？，self.fc1是一个实例化的对象，为什么具备了函数一样的用法，这是因为nn.Linear继承了nn.Module类，nn.Module类中定义了__call__方法。（可以ctrl不断进入来查看）\n",
    "\n",
    "在 Python 中，任何定义了 __call__ 方法的类，其实例都可以像函数一样被调用。\n",
    "\n",
    "当调用 self.fc1(x) 时，实际上执行的是：\n",
    "- self.fc1.__call__(x)（Python 的隐式调用）\n",
    "- 而 nn.Module 的 __call__ 方法会调用子类的 forward 方法（即 self.fc1.forward(x)）。这个方法就是个前向计算方法。\n",
    "\n",
    "relu是torch.relu()这个函数为了保持写法一致，又封装成了nn.ReLU()这个类。来保证接口的一致性\n",
    "\n",
    "\n",
    "PyTorch 官方强烈建议使用 self.fc1(x)，因为它会触发完整的前向传播流程（包括钩子函数）这是 PyTorch 的核心设计模式，几乎所有组件（如 nn.Conv2d、nn.ReLU、甚至整个模型）都可以这样调用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89c120a6",
   "metadata": {},
   "source": [
    "我们来介绍一下call方法是什么"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4f1983c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "# 不带参数的call方法\n",
    "class Counter:\n",
    "    def __init__(self):\n",
    "        self.count = 0\n",
    "    \n",
    "    def __call__(self):\n",
    "        self.count += 1\n",
    "        return self.count\n",
    "\n",
    "# 使用示例\n",
    "counter = Counter()\n",
    "print(counter())  # 输出: 1\n",
    "print(counter())  # 输出: 2\n",
    "print(counter.count)  # 输出: 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71fd2a18",
   "metadata": {},
   "source": [
    "类名后跟()，表示创建类的实例（对象），仅在第一次创建对象时发生。\n",
    "\n",
    "call方法无参数的情况下，在实例化之后，每次调用实例时触发 __call__ 方法\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "b36cf4b2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "唱跳篮球rap\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "# 带参数的call方法\n",
    "class Adder:\n",
    "    def __call__(self, a, b):\n",
    "        print(\"唱跳篮球rap\")\n",
    "        return a + b\n",
    "\n",
    "adder = Adder()\n",
    "print(adder(3, 5))  # 输出: 8"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "yolov5",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.20"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
